2023-04-24 18:53:29,900 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640 2023-04-24 18:53:29,913 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-24 18:53:29,928 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-24 18:53:29,929 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7, deleteOnExit=true 2023-04-24 18:53:29,929 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-24 18:53:29,930 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/test.cache.data in system properties and HBase conf 2023-04-24 18:53:29,930 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/hadoop.tmp.dir in system properties and HBase conf 2023-04-24 18:53:29,930 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/hadoop.log.dir in system properties and HBase conf 2023-04-24 18:53:29,931 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-24 18:53:29,932 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-24 18:53:29,932 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-24 18:53:30,052 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-24 18:53:30,471 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-24 18:53:30,475 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-24 18:53:30,475 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-24 18:53:30,475 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-24 18:53:30,476 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-24 18:53:30,476 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-24 18:53:30,476 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-24 18:53:30,476 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-24 18:53:30,477 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-24 18:53:30,477 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-24 18:53:30,477 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/nfs.dump.dir in system properties and HBase conf 2023-04-24 18:53:30,478 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/java.io.tmpdir in system properties and HBase conf 2023-04-24 18:53:30,478 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-24 18:53:30,478 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-24 18:53:30,479 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-24 18:53:31,102 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-24 18:53:31,107 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-24 18:53:31,387 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-24 18:53:31,580 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-24 18:53:31,598 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-24 18:53:31,636 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-24 18:53:31,667 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/java.io.tmpdir/Jetty_localhost_45169_hdfs____l3t7p9/webapp 2023-04-24 18:53:31,819 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45169 2023-04-24 18:53:31,856 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-24 18:53:31,857 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-24 18:53:32,280 WARN [Listener at localhost/36683] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-24 18:53:32,400 WARN [Listener at localhost/36683] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-24 18:53:32,432 WARN [Listener at localhost/36683] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-24 18:53:32,440 INFO [Listener at localhost/36683] log.Slf4jLog(67): jetty-6.1.26 2023-04-24 18:53:32,447 INFO [Listener at localhost/36683] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/java.io.tmpdir/Jetty_localhost_34573_datanode____8k80xj/webapp 2023-04-24 18:53:32,571 INFO [Listener at localhost/36683] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34573 2023-04-24 18:53:33,022 WARN [Listener at localhost/43631] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-24 18:53:33,604 DEBUG [Listener at localhost/43631] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640 2023-04-24 18:53:33,632 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xea1b44a9fe795a7c: Processing first storage report for DS-c342835e-435b-4edb-9be5-fe238aec49a3 from datanode 45dafbf2-b08d-4ede-9eaf-5017a3e0a134 2023-04-24 18:53:33,639 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xea1b44a9fe795a7c: from storage DS-c342835e-435b-4edb-9be5-fe238aec49a3 node DatanodeRegistration(127.0.0.1:43175, datanodeUuid=45dafbf2-b08d-4ede-9eaf-5017a3e0a134, infoPort=41857, infoSecurePort=0, ipcPort=43631, storageInfo=lv=-57;cid=testClusterID;nsid=1566391198;c=1682362411180), blocks: 0, hasStaleStorage: true, processing time: 6 msecs, invalidatedBlocks: 0 2023-04-24 18:53:33,639 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xea1b44a9fe795a7c: Processing first storage report for DS-7af034ec-061a-462b-b3b0-1d3b5de95a51 from datanode 45dafbf2-b08d-4ede-9eaf-5017a3e0a134 2023-04-24 18:53:33,639 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xea1b44a9fe795a7c: from storage DS-7af034ec-061a-462b-b3b0-1d3b5de95a51 node DatanodeRegistration(127.0.0.1:43175, datanodeUuid=45dafbf2-b08d-4ede-9eaf-5017a3e0a134, infoPort=41857, infoSecurePort=0, ipcPort=43631, storageInfo=lv=-57;cid=testClusterID;nsid=1566391198;c=1682362411180), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-24 18:53:33,696 INFO [Listener at localhost/43631] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7/zookeeper_0, clientPort=59583, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-24 18:53:33,711 INFO [Listener at localhost/43631] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59583 2023-04-24 18:53:33,721 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:33,723 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:34,381 INFO [Listener at localhost/43631] util.FSUtils(471): Created version file at hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed with version=8 2023-04-24 18:53:34,381 INFO [Listener at localhost/43631] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/hbase-staging 2023-04-24 18:53:34,671 INFO [Listener at localhost/43631] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-24 18:53:35,122 INFO [Listener at localhost/43631] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-24 18:53:35,153 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,154 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,154 INFO [Listener at localhost/43631] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-24 18:53:35,154 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,154 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-24 18:53:35,294 INFO [Listener at localhost/43631] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-24 18:53:35,370 DEBUG [Listener at localhost/43631] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-24 18:53:35,477 INFO [Listener at localhost/43631] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45309 2023-04-24 18:53:35,488 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:35,490 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:35,510 INFO [Listener at localhost/43631] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45309 connecting to ZooKeeper ensemble=127.0.0.1:59583 2023-04-24 18:53:35,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:453090x0, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-24 18:53:35,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45309-0x101b7851c360000 connected 2023-04-24 18:53:35,581 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 18:53:35,582 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 18:53:35,585 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-24 18:53:35,594 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45309 2023-04-24 18:53:35,594 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45309 2023-04-24 18:53:35,594 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45309 2023-04-24 18:53:35,595 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45309 2023-04-24 18:53:35,595 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45309 2023-04-24 18:53:35,602 INFO [Listener at localhost/43631] master.HMaster(444): hbase.rootdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed, hbase.cluster.distributed=false 2023-04-24 18:53:35,681 INFO [Listener at localhost/43631] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-24 18:53:35,681 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,681 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,681 INFO [Listener at localhost/43631] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-24 18:53:35,682 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 18:53:35,682 INFO [Listener at localhost/43631] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-24 18:53:35,687 INFO [Listener at localhost/43631] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-24 18:53:35,690 INFO [Listener at localhost/43631] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37593 2023-04-24 18:53:35,693 INFO [Listener at localhost/43631] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-24 18:53:35,699 DEBUG [Listener at localhost/43631] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-24 18:53:35,700 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:35,703 INFO [Listener at localhost/43631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:35,706 INFO [Listener at localhost/43631] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37593 connecting to ZooKeeper ensemble=127.0.0.1:59583 2023-04-24 18:53:35,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:375930x0, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-24 18:53:35,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37593-0x101b7851c360001 connected 2023-04-24 18:53:35,711 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 18:53:35,712 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 18:53:35,713 DEBUG [Listener at localhost/43631] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-24 18:53:35,714 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37593 2023-04-24 18:53:35,714 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37593 2023-04-24 18:53:35,714 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37593 2023-04-24 18:53:35,718 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37593 2023-04-24 18:53:35,718 DEBUG [Listener at localhost/43631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37593 2023-04-24 18:53:35,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:35,736 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-24 18:53:35,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:35,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-24 18:53:35,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-24 18:53:35,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:35,757 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-24 18:53:35,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45309,1682362414518 from backup master directory 2023-04-24 18:53:35,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-24 18:53:35,762 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:35,762 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-24 18:53:35,763 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-24 18:53:35,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:35,766 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-24 18:53:35,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-24 18:53:35,851 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/hbase.id with ID: 01b9d6fb-5d58-4347-a1a0-6d65a3d301ec 2023-04-24 18:53:35,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 18:53:35,906 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:35,945 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x14082f89 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:35,979 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4664f591, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:36,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:53:36,006 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-24 18:53:36,027 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-24 18:53:36,027 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-24 18:53:36,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:53:36,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:53:36,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 18:53:36,071 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store-tmp 2023-04-24 18:53:36,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:36,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-24 18:53:36,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:53:36,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:53:36,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-24 18:53:36,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:53:36,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:53:36,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 18:53:36,102 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/WALs/jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:36,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45309%2C1682362414518, suffix=, logDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/WALs/jenkins-hbase4.apache.org,45309,1682362414518, archiveDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/oldWALs, maxLogs=10 2023-04-24 18:53:36,179 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK] 2023-04-24 18:53:36,188 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:53:36,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/WALs/jenkins-hbase4.apache.org,45309,1682362414518/jenkins-hbase4.apache.org%2C45309%2C1682362414518.1682362416134 2023-04-24 18:53:36,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK]] 2023-04-24 18:53:36,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:53:36,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:36,264 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,265 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,325 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,333 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-24 18:53:36,361 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-24 18:53:36,375 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:36,382 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 18:53:36,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:53:36,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59953815, jitterRate=-0.10661853849887848}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-24 18:53:36,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 18:53:36,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-24 18:53:36,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-24 18:53:36,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-24 18:53:36,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-24 18:53:36,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-24 18:53:36,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-04-24 18:53:36,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-24 18:53:36,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-24 18:53:36,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-24 18:53:36,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-24 18:53:36,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-24 18:53:36,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-24 18:53:36,545 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-24 18:53:36,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-24 18:53:36,555 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:36,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-24 18:53:36,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-24 18:53:36,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-24 18:53:36,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-24 18:53:36,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-24 18:53:36,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:36,574 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,45309,1682362414518, sessionid=0x101b7851c360000, setting cluster-up flag (Was=false) 2023-04-24 18:53:36,592 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:36,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-24 18:53:36,600 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:36,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:36,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-24 18:53:36,614 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:36,617 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.hbase-snapshot/.tmp 2023-04-24 18:53:36,622 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(952): ClusterId : 01b9d6fb-5d58-4347-a1a0-6d65a3d301ec 2023-04-24 18:53:36,628 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-24 18:53:36,634 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-24 18:53:36,634 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-24 18:53:36,638 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-24 18:53:36,639 DEBUG [RS:0;jenkins-hbase4:37593] zookeeper.ReadOnlyZKClient(139): Connect 0x612da705 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:36,647 DEBUG [RS:0;jenkins-hbase4:37593] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fea5550, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:36,648 DEBUG [RS:0;jenkins-hbase4:37593] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bd5cc75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-24 18:53:36,675 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:37593 2023-04-24 18:53:36,679 INFO [RS:0;jenkins-hbase4:37593] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-24 18:53:36,679 INFO [RS:0;jenkins-hbase4:37593] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-24 18:53:36,680 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1023): About to register with Master. 2023-04-24 18:53:36,682 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45309,1682362414518 with isa=jenkins-hbase4.apache.org/172.31.14.131:37593, startcode=1682362415680 2023-04-24 18:53:36,700 DEBUG [RS:0;jenkins-hbase4:37593] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-24 18:53:36,756 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-24 18:53:36,756 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48109, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-24 18:53:36,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:36,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-24 18:53:36,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-24 18:53:36,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-24 18:53:36,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-24 18:53:36,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-24 18:53:36,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:36,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-24 18:53:36,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:36,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682362446772 2023-04-24 18:53:36,775 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-24 18:53:36,779 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-24 18:53:36,780 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-24 18:53:36,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-24 18:53:36,790 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-24 18:53:36,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-24 18:53:36,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-24 18:53:36,798 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-24 18:53:36,798 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-24 18:53:36,804 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:36,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-24 18:53:36,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-24 18:53:36,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-24 18:53:36,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-24 18:53:36,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-24 18:53:36,812 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682362416811,5,FailOnTimeoutGroup] 2023-04-24 18:53:36,812 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682362416812,5,FailOnTimeoutGroup] 2023-04-24 18:53:36,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:36,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-24 18:53:36,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:36,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:36,815 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(2829): Master is not running yet 2023-04-24 18:53:36,816 WARN [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-24 18:53:36,917 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45309,1682362414518 with isa=jenkins-hbase4.apache.org/172.31.14.131:37593, startcode=1682362415680 2023-04-24 18:53:36,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:36,932 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed 2023-04-24 18:53:36,932 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36683 2023-04-24 18:53:36,932 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-24 18:53:36,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 18:53:36,940 DEBUG [RS:0;jenkins-hbase4:37593] zookeeper.ZKUtil(162): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:36,940 WARN [RS:0;jenkins-hbase4:37593] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-24 18:53:36,940 INFO [RS:0;jenkins-hbase4:37593] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 18:53:36,941 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/WALs/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:36,941 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37593,1682362415680] 2023-04-24 18:53:36,950 DEBUG [RS:0;jenkins-hbase4:37593] zookeeper.ZKUtil(162): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:36,964 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-24 18:53:36,975 INFO [RS:0;jenkins-hbase4:37593] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-24 18:53:37,003 INFO [RS:0;jenkins-hbase4:37593] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-24 18:53:37,008 INFO [RS:0;jenkins-hbase4:37593] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-24 18:53:37,008 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,010 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-24 18:53:37,018 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,018 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,019 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,019 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,019 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,019 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,020 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-24 18:53:37,020 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,020 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,020 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,020 DEBUG [RS:0;jenkins-hbase4:37593] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-24 18:53:37,021 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,022 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,022 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,041 INFO [RS:0;jenkins-hbase4:37593] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-24 18:53:37,043 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37593,1682362415680-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,059 INFO [RS:0;jenkins-hbase4:37593] regionserver.Replication(203): jenkins-hbase4.apache.org,37593,1682362415680 started 2023-04-24 18:53:37,059 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37593,1682362415680, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37593, sessionid=0x101b7851c360001 2023-04-24 18:53:37,060 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-24 18:53:37,060 DEBUG [RS:0;jenkins-hbase4:37593] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:37,060 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37593,1682362415680' 2023-04-24 18:53:37,060 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37593,1682362415680' 2023-04-24 18:53:37,061 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-24 18:53:37,062 DEBUG [RS:0;jenkins-hbase4:37593] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-24 18:53:37,062 DEBUG [RS:0;jenkins-hbase4:37593] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-24 18:53:37,062 INFO [RS:0;jenkins-hbase4:37593] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-24 18:53:37,062 INFO [RS:0;jenkins-hbase4:37593] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-24 18:53:37,174 INFO [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37593%2C1682362415680, suffix=, logDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/WALs/jenkins-hbase4.apache.org,37593,1682362415680, archiveDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/oldWALs, maxLogs=32 2023-04-24 18:53:37,192 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK] 2023-04-24 18:53:37,196 INFO [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/WALs/jenkins-hbase4.apache.org,37593,1682362415680/jenkins-hbase4.apache.org%2C37593%2C1682362415680.1682362417176 2023-04-24 18:53:37,196 DEBUG [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK]] 2023-04-24 18:53:37,226 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-24 18:53:37,228 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-24 18:53:37,228 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed 2023-04-24 18:53:37,242 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:37,245 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-24 18:53:37,248 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/info 2023-04-24 18:53:37,250 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-24 18:53:37,251 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,251 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-24 18:53:37,254 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/rep_barrier 2023-04-24 18:53:37,255 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-24 18:53:37,256 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,256 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-24 18:53:37,259 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/table 2023-04-24 18:53:37,260 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-24 18:53:37,261 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,263 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740 2023-04-24 18:53:37,264 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740 2023-04-24 18:53:37,269 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:53:37,271 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-24 18:53:37,275 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:53:37,276 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70146996, jitterRate=0.04527169466018677}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:53:37,276 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-24 18:53:37,276 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-24 18:53:37,276 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-24 18:53:37,276 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-24 18:53:37,277 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-24 18:53:37,277 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-24 18:53:37,278 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-24 18:53:37,278 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-24 18:53:37,284 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-24 18:53:37,284 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-24 18:53:37,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-24 18:53:37,305 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-24 18:53:37,308 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-24 18:53:37,461 DEBUG [jenkins-hbase4:45309] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-24 18:53:37,465 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37593,1682362415680, state=OPENING 2023-04-24 18:53:37,478 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-24 18:53:37,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:37,481 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-24 18:53:37,483 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:53:37,671 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:37,673 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-24 18:53:37,678 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-24 18:53:37,693 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-24 18:53:37,694 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 18:53:37,699 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37593%2C1682362415680.meta, suffix=.meta, logDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/WALs/jenkins-hbase4.apache.org,37593,1682362415680, archiveDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/oldWALs, maxLogs=32 2023-04-24 18:53:37,720 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK] 2023-04-24 18:53:37,724 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/WALs/jenkins-hbase4.apache.org,37593,1682362415680/jenkins-hbase4.apache.org%2C37593%2C1682362415680.meta.1682362417701.meta 2023-04-24 18:53:37,728 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43175,DS-c342835e-435b-4edb-9be5-fe238aec49a3,DISK]] 2023-04-24 18:53:37,728 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:53:37,730 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-24 18:53:37,749 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-24 18:53:37,754 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-24 18:53:37,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-24 18:53:37,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:37,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-24 18:53:37,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-24 18:53:37,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-24 18:53:37,767 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/info 2023-04-24 18:53:37,767 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/info 2023-04-24 18:53:37,768 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-24 18:53:37,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-24 18:53:37,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/rep_barrier 2023-04-24 18:53:37,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/rep_barrier 2023-04-24 18:53:37,771 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-24 18:53:37,772 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,772 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-24 18:53:37,773 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/table 2023-04-24 18:53:37,773 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/table 2023-04-24 18:53:37,774 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-24 18:53:37,775 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:37,777 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740 2023-04-24 18:53:37,780 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740 2023-04-24 18:53:37,783 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:53:37,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-24 18:53:37,788 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60561365, jitterRate=-0.09756533801555634}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:53:37,788 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-24 18:53:37,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682362417663 2023-04-24 18:53:37,814 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-24 18:53:37,814 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-24 18:53:37,815 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37593,1682362415680, state=OPEN 2023-04-24 18:53:37,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-24 18:53:37,818 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-24 18:53:37,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-24 18:53:37,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37593,1682362415680 in 335 msec 2023-04-24 18:53:37,828 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-24 18:53:37,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 531 msec 2023-04-24 18:53:37,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1600 sec 2023-04-24 18:53:37,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682362417835, completionTime=-1 2023-04-24 18:53:37,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-24 18:53:37,836 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-24 18:53:37,908 DEBUG [hconnection-0x28481538-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:37,912 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:37,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-24 18:53:37,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682362477930 2023-04-24 18:53:37,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682362537930 2023-04-24 18:53:37,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 94 msec 2023-04-24 18:53:37,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45309,1682362414518-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45309,1682362414518-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,957 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45309,1682362414518-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,960 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45309, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,961 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-24 18:53:37,968 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-24 18:53:37,976 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-24 18:53:37,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-24 18:53:37,989 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-24 18:53:37,992 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:53:37,994 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:53:38,017 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,020 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed empty. 2023-04-24 18:53:38,020 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,021 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-24 18:53:38,450 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-24 18:53:38,453 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4e0aadbf319324c8ff606d0c7af7f8ed, NAME => 'hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:53:38,464 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:38,464 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4e0aadbf319324c8ff606d0c7af7f8ed, disabling compactions & flushes 2023-04-24 18:53:38,464 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,464 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,464 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. after waiting 0 ms 2023-04-24 18:53:38,464 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,464 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,465 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4e0aadbf319324c8ff606d0c7af7f8ed: 2023-04-24 18:53:38,469 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:53:38,484 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682362418472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362418472"}]},"ts":"1682362418472"} 2023-04-24 18:53:38,510 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:53:38,511 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:53:38,516 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362418512"}]},"ts":"1682362418512"} 2023-04-24 18:53:38,520 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-24 18:53:38,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4e0aadbf319324c8ff606d0c7af7f8ed, ASSIGN}] 2023-04-24 18:53:38,532 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4e0aadbf319324c8ff606d0c7af7f8ed, ASSIGN 2023-04-24 18:53:38,533 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4e0aadbf319324c8ff606d0c7af7f8ed, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:53:38,684 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4e0aadbf319324c8ff606d0c7af7f8ed, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:38,685 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682362418684"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362418684"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362418684"}]},"ts":"1682362418684"} 2023-04-24 18:53:38,689 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4e0aadbf319324c8ff606d0c7af7f8ed, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:53:38,848 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,849 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4e0aadbf319324c8ff606d0c7af7f8ed, NAME => 'hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:53:38,850 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,850 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:38,850 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,851 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,853 INFO [StoreOpener-4e0aadbf319324c8ff606d0c7af7f8ed-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,855 DEBUG [StoreOpener-4e0aadbf319324c8ff606d0c7af7f8ed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/info 2023-04-24 18:53:38,855 DEBUG [StoreOpener-4e0aadbf319324c8ff606d0c7af7f8ed-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/info 2023-04-24 18:53:38,856 INFO [StoreOpener-4e0aadbf319324c8ff606d0c7af7f8ed-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e0aadbf319324c8ff606d0c7af7f8ed columnFamilyName info 2023-04-24 18:53:38,857 INFO [StoreOpener-4e0aadbf319324c8ff606d0c7af7f8ed-1] regionserver.HStore(310): Store=4e0aadbf319324c8ff606d0c7af7f8ed/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:38,858 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,859 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:53:38,870 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:53:38,871 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4e0aadbf319324c8ff606d0c7af7f8ed; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63324994, jitterRate=-0.05638405680656433}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-24 18:53:38,871 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4e0aadbf319324c8ff606d0c7af7f8ed: 2023-04-24 18:53:38,873 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed., pid=6, masterSystemTime=1682362418842 2023-04-24 18:53:38,878 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,878 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:53:38,880 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4e0aadbf319324c8ff606d0c7af7f8ed, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:38,881 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682362418879"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362418879"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362418879"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362418879"}]},"ts":"1682362418879"} 2023-04-24 18:53:38,889 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-24 18:53:38,889 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4e0aadbf319324c8ff606d0c7af7f8ed, server=jenkins-hbase4.apache.org,37593,1682362415680 in 195 msec 2023-04-24 18:53:38,893 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-24 18:53:38,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4e0aadbf319324c8ff606d0c7af7f8ed, ASSIGN in 360 msec 2023-04-24 18:53:38,895 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:53:38,896 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362418895"}]},"ts":"1682362418895"} 2023-04-24 18:53:38,899 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-24 18:53:38,903 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:53:38,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 924 msec 2023-04-24 18:53:38,992 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-24 18:53:38,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-24 18:53:38,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:53:39,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-24 18:53:39,051 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-24 18:53:39,057 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-24 18:53:39,069 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-24 18:53:39,083 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-24 18:53:39,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-04-24 18:53:39,105 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-24 18:53:39,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-24 18:53:39,110 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.347sec 2023-04-24 18:53:39,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-24 18:53:39,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-24 18:53:39,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-24 18:53:39,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45309,1682362414518-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-24 18:53:39,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45309,1682362414518-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-24 18:53:39,126 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-24 18:53:39,128 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x5e324982 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:39,133 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373f9c99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:39,145 DEBUG [hconnection-0x258148f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:39,158 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:39,168 INFO [Listener at localhost/43631] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:53:39,192 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=352, ProcessCount=170, AvailableMemoryMB=5176 2023-04-24 18:53:39,209 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:53:39,213 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:53:39,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:53:39,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:53:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:53:39,233 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:53:39,235 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:53:39,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-24 18:53:39,238 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:39,240 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 empty. 2023-04-24 18:53:39,243 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:39,243 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:53:39,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 18:53:39,262 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:53:39,265 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 84739daff3fefab14f3952d64d6f3630, NAME => 'TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:53:39,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 18:53:39,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 18:53:39,679 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:39,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 84739daff3fefab14f3952d64d6f3630, disabling compactions & flushes 2023-04-24 18:53:39,680 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:39,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:39,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. after waiting 0 ms 2023-04-24 18:53:39,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:39,680 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:39,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:39,686 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:53:39,687 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362419687"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362419687"}]},"ts":"1682362419687"} 2023-04-24 18:53:39,690 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:53:39,692 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:53:39,692 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362419692"}]},"ts":"1682362419692"} 2023-04-24 18:53:39,695 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:53:39,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, ASSIGN}] 2023-04-24 18:53:39,704 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, ASSIGN 2023-04-24 18:53:39,706 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:53:39,857 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=84739daff3fefab14f3952d64d6f3630, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:39,858 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362419857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362419857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362419857"}]},"ts":"1682362419857"} 2023-04-24 18:53:39,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:53:39,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 18:53:40,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:40,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 84739daff3fefab14f3952d64d6f3630, NAME => 'TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:53:40,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:53:40,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,023 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,026 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A 2023-04-24 18:53:40,026 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A 2023-04-24 18:53:40,027 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:53:40,028 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 84739daff3fefab14f3952d64d6f3630 columnFamilyName A 2023-04-24 18:53:40,029 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(310): Store=84739daff3fefab14f3952d64d6f3630/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:40,029 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,031 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B 2023-04-24 18:53:40,031 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B 2023-04-24 18:53:40,031 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:53:40,031 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 84739daff3fefab14f3952d64d6f3630 columnFamilyName B 2023-04-24 18:53:40,032 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(310): Store=84739daff3fefab14f3952d64d6f3630/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:40,032 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,034 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C 2023-04-24 18:53:40,034 DEBUG [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C 2023-04-24 18:53:40,034 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:53:40,035 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 84739daff3fefab14f3952d64d6f3630 columnFamilyName C 2023-04-24 18:53:40,035 INFO [StoreOpener-84739daff3fefab14f3952d64d6f3630-1] regionserver.HStore(310): Store=84739daff3fefab14f3952d64d6f3630/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:53:40,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:40,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:53:40,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:53:40,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 84739daff3fefab14f3952d64d6f3630; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66906987, jitterRate=-0.0030082017183303833}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:53:40,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:40,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., pid=11, masterSystemTime=1682362420015 2023-04-24 18:53:40,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:40,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:40,053 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=84739daff3fefab14f3952d64d6f3630, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,053 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362420053"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362420053"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362420053"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362420053"}]},"ts":"1682362420053"} 2023-04-24 18:53:40,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-24 18:53:40,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 in 195 msec 2023-04-24 18:53:40,061 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-24 18:53:40,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, ASSIGN in 356 msec 2023-04-24 18:53:40,063 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:53:40,063 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362420063"}]},"ts":"1682362420063"} 2023-04-24 18:53:40,065 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:53:40,068 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:53:40,071 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 840 msec 2023-04-24 18:53:40,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 18:53:40,366 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-24 18:53:40,371 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x55e5fd59 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,378 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4281aac2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,380 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,389 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,392 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:53:40,404 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:53:40,414 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,419 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,419 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,427 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,428 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,436 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,436 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,441 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,441 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x60b8c4c7 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,446 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49a45b6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,448 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,453 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,454 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,476 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,477 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,482 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,482 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1423ce11 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:53:40,487 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fab6dfa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:53:40,494 DEBUG [hconnection-0x6fed7337-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,495 DEBUG [hconnection-0x4d06a665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,495 DEBUG [hconnection-0x1516ed88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,504 DEBUG [hconnection-0x9312249-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,506 DEBUG [hconnection-0x1531940c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,511 DEBUG [hconnection-0x17c8a7bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,522 DEBUG [hconnection-0x74b555f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,525 DEBUG [hconnection-0x66fde656-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,525 DEBUG [hconnection-0x637b85c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:53:40,529 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,533 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,533 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,533 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,537 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,538 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35468, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,541 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,544 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:40,550 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:53:40,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:40,596 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:40,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:40,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:40,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:40,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:40,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:40,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:40,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:40,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:40,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:40,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:40,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:40,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:40,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:40,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:40,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:40,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:40,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:40,628 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:40,634 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:40,636 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:53:40,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:40,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:40,665 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:40,665 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:40,665 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:40,665 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:40,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:40,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362480767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362480768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362420739, max timestamp=1682362420766Num uniques 859; 2023-04-24 18:53:40,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:40,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362420739, max timestamp=1682362420766Num uniques 859; 2023-04-24 18:53:40,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:40,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362420739, max timestamp=1682362420762Num uniques 859; 2023-04-24 18:53:40,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362480863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362420836, max timestamp=1682362420860Num uniques 1018; 2023-04-24 18:53:40,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:40,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362420836, max timestamp=1682362420864Num uniques 1018; 2023-04-24 18:53:40,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:40,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362420836, max timestamp=1682362420864Num uniques 1018; 2023-04-24 18:53:40,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:40,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362480968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362480972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362480987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:40,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682362420917, max timestamp=1682362420971Num uniques 1027; 2023-04-24 18:53:40,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-24 18:53:40,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682362420917, max timestamp=1682362420970Num uniques 1027; 2023-04-24 18:53:40,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-24 18:53:40,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10386, totalHeapSize=23848, min timestamp=1682362420917, max timestamp=1682362420971Num uniques 1027; 2023-04-24 18:53:40,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47026, new segment data size=10386, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:41,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362421056, max timestamp=1682362421076Num uniques 1086; 2023-04-24 18:53:41,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:53:41,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362421056, max timestamp=1682362421076Num uniques 1009; 2023-04-24 18:53:41,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-24 18:53:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362421056, max timestamp=1682362421076Num uniques 1009; 2023-04-24 18:53:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-24 18:53:41,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362481111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362421110, max timestamp=1682362421116Num uniques 1045; 2023-04-24 18:53:41,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362481114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:53:41,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362421110, max timestamp=1682362421116Num uniques 1045; 2023-04-24 18:53:41,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:53:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362421110, max timestamp=1682362421116Num uniques 1045; 2023-04-24 18:53:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:53:41,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362481162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362481163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362481165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362481222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362481224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362481268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362481269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362481269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,296 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/7877532514be4bfe860cba1edb07cff2 2023-04-24 18:53:41,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362481428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362481429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362481476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362481478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362481478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362481734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362481734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362481783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362481787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362481787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,813 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/7db4cfd9b07f426dbd6400ca79af518b 2023-04-24 18:53:41,854 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/9c89e84ec7e5411d8fe6f778ca7345cf 2023-04-24 18:53:41,868 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/7877532514be4bfe860cba1edb07cff2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2 2023-04-24 18:53:41,881 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2, entries=150, sequenceid=10, filesize=11.6 K 2023-04-24 18:53:41,889 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/7db4cfd9b07f426dbd6400ca79af518b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b 2023-04-24 18:53:41,913 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b, entries=150, sequenceid=10, filesize=11.6 K 2023-04-24 18:53:41,916 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/9c89e84ec7e5411d8fe6f778ca7345cf as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf 2023-04-24 18:53:41,937 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf, entries=150, sequenceid=10, filesize=11.6 K 2023-04-24 18:53:41,940 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 84739daff3fefab14f3952d64d6f3630 in 1304ms, sequenceid=10, compaction requested=false 2023-04-24 18:53:41,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-24 18:53:41,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:41,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:41,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:41,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:41,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:41,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:41,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b26aa60[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:41,974 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,974 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,974 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:41,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:41,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:41,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:41,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:41,998 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:41,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:41,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:41,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:41,998 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:42,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:42,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:42,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:42,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:42,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:42,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:42,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:42,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:42,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:42,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:42,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:42,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:42,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:42,173 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:42,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:42,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:42,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:42,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:42,178 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:42,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:42,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:42,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:42,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:42,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:42,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:42,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:42,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:42,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:42,194 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:42,195 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:42,197 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:42,199 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=140.89 KB heapSize=401.53 KB 2023-04-24 18:53:42,199 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:42,201 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:42,201 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:42,201 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:42,201 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:42,201 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:42,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:42,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:42,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362482285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362482285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362482288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362482291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362482293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362482394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362482395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362482600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:42,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362482602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,654 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/9b28a2e214244347a861ddccd3d3b254 2023-04-24 18:53:42,716 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/e80f3f1c41dc4ca7b321371814a228eb 2023-04-24 18:53:42,821 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3c93efaeab4644889d832bb4a5b01193 2023-04-24 18:53:42,850 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/9b28a2e214244347a861ddccd3d3b254 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254 2023-04-24 18:53:42,867 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 18:53:42,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/e80f3f1c41dc4ca7b321371814a228eb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb 2023-04-24 18:53:42,905 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 18:53:42,911 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3c93efaeab4644889d832bb4a5b01193 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193 2023-04-24 18:53:42,934 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 18:53:42,937 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~401.48 KB/411120, currentSize=40.25 KB/41220 for 84739daff3fefab14f3952d64d6f3630 in 739ms, sequenceid=116, compaction requested=false 2023-04-24 18:53:42,937 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:42,937 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:42,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:42,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:42,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:53:42,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:42,947 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,947 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:42,947 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:42,947 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:42,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:42,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:42,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:42,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d66652[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:42,950 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:42,950 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,952 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:42,953 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:42,953 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:42,953 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:42,972 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:42,973 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:42,989 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:42,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:42,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:42,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:42,989 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:42,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:42,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:42,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:42,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:43,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:43,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:43,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,018 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:43,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:43,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:43,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:43,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:43,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:43,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:43,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/84f8f0ceee07439988c1ca6c4d797e4e 2023-04-24 18:53:43,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362423028, max timestamp=1682362423093Num uniques 859; 2023-04-24 18:53:43,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:43,123 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:43,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362423028, max timestamp=1682362423094Num uniques 859; 2023-04-24 18:53:43,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362423028, max timestamp=1682362423094Num uniques 859; 2023-04-24 18:53:43,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:43,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:43,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:43,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:43,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f6d6e84508ae4d69acd82684a9ba9e3c 2023-04-24 18:53:43,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:43,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:43,152 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:43,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:43,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:43,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:43,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,163 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:43,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:43,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:43,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:43,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:43,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:43,200 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:43,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:43,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15db9014[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:43,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:43,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,215 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,215 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:43,215 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:43,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:43,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:43,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362483214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362483232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:43,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:43,244 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:43,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:43,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:43,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:43,245 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:43,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,259 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7920f0b06920477983e1bc775039ab7c 2023-04-24 18:53:43,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/84f8f0ceee07439988c1ca6c4d797e4e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e 2023-04-24 18:53:43,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:43,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:43,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:43,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:43,284 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:43,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:43,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:43,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362483298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362483302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e, entries=200, sequenceid=127, filesize=13.9 K 2023-04-24 18:53:43,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f6d6e84508ae4d69acd82684a9ba9e3c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c 2023-04-24 18:53:43,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362483307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,333 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:53:43,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c, entries=150, sequenceid=127, filesize=11.6 K 2023-04-24 18:53:43,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362483336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362483338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7920f0b06920477983e1bc775039ab7c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c 2023-04-24 18:53:43,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c, entries=150, sequenceid=127, filesize=11.6 K 2023-04-24 18:53:43,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 84739daff3fefab14f3952d64d6f3630 in 449ms, sequenceid=127, compaction requested=true 2023-04-24 18:53:43,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:43,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:43,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:43,402 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:43,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:43,403 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:43,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:43,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:43,408 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:43,408 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:43,410 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:43,410 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:43,411 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,411 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,411 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=37.1 K 2023-04-24 18:53:43,411 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=34.8 K 2023-04-24 18:53:43,412 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7877532514be4bfe860cba1edb07cff2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682362420586 2023-04-24 18:53:43,412 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7db4cfd9b07f426dbd6400ca79af518b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682362420586 2023-04-24 18:53:43,423 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9b28a2e214244347a861ddccd3d3b254, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682362421154 2023-04-24 18:53:43,423 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e80f3f1c41dc4ca7b321371814a228eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682362421154 2023-04-24 18:53:43,424 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f6d6e84508ae4d69acd82684a9ba9e3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422275 2023-04-24 18:53:43,424 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 84f8f0ceee07439988c1ca6c4d797e4e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422271 2023-04-24 18:53:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:43,427 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:43,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:43,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:43,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:43,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:43,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:43,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:43,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:43,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:43,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:43,444 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 18:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:43,496 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-24 18:53:43,497 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-24 18:53:43,499 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#10 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:43,503 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#11 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:43,556 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/fba8f3f1a87b4064808d695e958d7889 2023-04-24 18:53:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:43,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:43,576 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/82c1c1de02c84f8aaea9da8814e336f3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/82c1c1de02c84f8aaea9da8814e336f3 2023-04-24 18:53:43,599 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 82c1c1de02c84f8aaea9da8814e336f3(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:43,600 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:43,600 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362423401; duration=0sec 2023-04-24 18:53:43,604 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:43,604 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:43,607 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:43,608 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:43,608 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,608 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=34.8 K 2023-04-24 18:53:43,609 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9c89e84ec7e5411d8fe6f778ca7345cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682362420586 2023-04-24 18:53:43,612 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3c93efaeab4644889d832bb4a5b01193, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682362421154 2023-04-24 18:53:43,613 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7920f0b06920477983e1bc775039ab7c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422275 2023-04-24 18:53:43,641 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#13 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:43,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362483633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362483644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,654 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/18db87669eb340b19beca1a5107ee3a4 2023-04-24 18:53:43,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362483776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:43,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362483776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,791 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/4d43ab62d4554c7389ebcf51ceeb177a 2023-04-24 18:53:43,808 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/fba8f3f1a87b4064808d695e958d7889 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889 2023-04-24 18:53:43,820 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889, entries=150, sequenceid=163, filesize=11.8 K 2023-04-24 18:53:43,827 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/18db87669eb340b19beca1a5107ee3a4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4 2023-04-24 18:53:43,843 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4, entries=150, sequenceid=163, filesize=11.8 K 2023-04-24 18:53:43,846 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/4d43ab62d4554c7389ebcf51ceeb177a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a 2023-04-24 18:53:43,855 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a, entries=150, sequenceid=163, filesize=11.8 K 2023-04-24 18:53:43,856 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 84739daff3fefab14f3952d64d6f3630 in 412ms, sequenceid=163, compaction requested=false 2023-04-24 18:53:43,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:43,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:43,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,865 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:43,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7727936c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:43,870 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:43,870 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,871 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:43,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:43,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:43,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:43,885 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:43,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:43,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:43,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:43,886 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:43,889 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:43,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:43,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,891 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:43,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:43,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:43,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:43,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:43,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:43,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:43,920 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:43,920 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:43,972 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/ce55959d03fe4287b2e9caf366f06c68 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ce55959d03fe4287b2e9caf366f06c68 2023-04-24 18:53:43,987 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into ce55959d03fe4287b2e9caf366f06c68(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:43,987 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:43,987 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362423401; duration=0sec 2023-04-24 18:53:43,987 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:43,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:43,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:53:43,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:43,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:43,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:43,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:43,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:43,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:44,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:44,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:44,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:44,051 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:44,051 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:44,052 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:44,053 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:44,061 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/c5e5667307464b3aab3e1d7a3d549e8e 2023-04-24 18:53:44,069 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:44,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:44,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:44,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:44,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:44,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:44,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:44,097 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:44,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:44,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:44,111 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/866a4e271b6b4b8ab6629c0544743333 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/866a4e271b6b4b8ab6629c0544743333 2023-04-24 18:53:44,125 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 866a4e271b6b4b8ab6629c0544743333(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:44,125 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:44,125 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362423403; duration=0sec 2023-04-24 18:53:44,126 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:44,129 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,129 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:44,129 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:44,129 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:44,142 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@193c4242[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:44,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:44,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,148 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:44,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:44,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:44,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:44,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:44,155 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:44,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:44,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:44,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:44,155 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362484140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362484141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:44,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:44,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,185 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,185 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,185 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:44,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:44,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:44,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:44,185 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:44,187 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,186 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:44,187 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:44,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:44,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:44,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/3970df994e3043df880acfe1e3b8d8e3 2023-04-24 18:53:44,197 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:44,198 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:44,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17 2023-04-24 18:53:44,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/c5e5667307464b3aab3e1d7a3d549e8e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e 2023-04-24 18:53:44,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e, entries=200, sequenceid=177, filesize=14.1 K 2023-04-24 18:53:44,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/3970df994e3043df880acfe1e3b8d8e3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3 2023-04-24 18:53:44,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:44,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35456 deadline: 1682362484268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35498 deadline: 1682362484275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3, entries=150, sequenceid=177, filesize=11.8 K 2023-04-24 18:53:44,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17 2023-04-24 18:53:44,298 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:44,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:44,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17, entries=150, sequenceid=177, filesize=11.8 K 2023-04-24 18:53:44,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:44,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 84739daff3fefab14f3952d64d6f3630 in 334ms, sequenceid=177, compaction requested=true 2023-04-24 18:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:44,328 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:44,328 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:44,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:44,332 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38451 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:44,332 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:44,332 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,332 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ce55959d03fe4287b2e9caf366f06c68, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=37.5 K 2023-04-24 18:53:44,333 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:44,333 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:44,333 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,333 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/82c1c1de02c84f8aaea9da8814e336f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=35.2 K 2023-04-24 18:53:44,334 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting ce55959d03fe4287b2e9caf366f06c68, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422275 2023-04-24 18:53:44,334 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 82c1c1de02c84f8aaea9da8814e336f3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422275 2023-04-24 18:53:44,355 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fba8f3f1a87b4064808d695e958d7889, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682362423156 2023-04-24 18:53:44,355 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 18db87669eb340b19beca1a5107ee3a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682362423156 2023-04-24 18:53:44,356 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c5e5667307464b3aab3e1d7a3d549e8e, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:44,356 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3970df994e3043df880acfe1e3b8d8e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:44,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:44,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:44,372 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:44,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:44,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:44,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:44,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:44,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:44,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:44,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:44,392 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#18 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:44,402 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#19 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:44,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:44,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:44,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:44,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:44,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424051, max timestamp=1682362424478Num uniques 859; 2023-04-24 18:53:44,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,491 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,491 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:44,492 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=82.74 KB heapSize=241.95 KB 2023-04-24 18:53:44,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:44,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424051, max timestamp=1682362424478Num uniques 859; 2023-04-24 18:53:44,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:44,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424051, max timestamp=1682362424478Num uniques 859; 2023-04-24 18:53:44,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:44,577 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/2f5fa87b883a4c69bb30d27b9230493a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2f5fa87b883a4c69bb30d27b9230493a 2023-04-24 18:53:44,598 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 2f5fa87b883a4c69bb30d27b9230493a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:44,598 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:44,598 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362424328; duration=0sec 2023-04-24 18:53:44,598 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:44,598 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:44,601 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:44,601 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:44,601 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:44,601 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/866a4e271b6b4b8ab6629c0544743333, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=35.2 K 2023-04-24 18:53:44,608 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 866a4e271b6b4b8ab6629c0544743333, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362422275 2023-04-24 18:53:44,609 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/03db3d6ecd2941c780c188d336a6f983 2023-04-24 18:53:44,609 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4d43ab62d4554c7389ebcf51ceeb177a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682362423156 2023-04-24 18:53:44,614 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7a7bdb00ea8b4966b8e0b91ac2fa4f17, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:44,644 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:44,671 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-24 18:53:44,672 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424601, max timestamp=1682362424652Num uniques 859; 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424601, max timestamp=1682362424652Num uniques 859; 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362424601, max timestamp=1682362424652Num uniques 859; 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362424735, max timestamp=1682362424762Num uniques 1018; 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362424735, max timestamp=1682362424762Num uniques 1018; 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362424735, max timestamp=1682362424760Num uniques 1018; 2023-04-24 18:53:44,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362424812, max timestamp=1682362424829Num uniques 1027; 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362424812, max timestamp=1682362424829Num uniques 1027; 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362424812, max timestamp=1682362424829Num uniques 1027; 2023-04-24 18:53:44,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362424868, max timestamp=1682362424890Num uniques 1086; 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362424868, max timestamp=1682362424890Num uniques 1086; 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362424868, max timestamp=1682362424890Num uniques 1086; 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:53:44,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:53:44,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682362424917, max timestamp=1682362424938Num uniques 1145; 2023-04-24 18:53:44,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362424926, max timestamp=1682362424938Num uniques 1145; 2023-04-24 18:53:44,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 18:53:44,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-24 18:53:44,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362424926, max timestamp=1682362424938Num uniques 1145; 2023-04-24 18:53:44,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 18:53:44,949 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/8efb055dff3b47a0b6515492fcfdc8d0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8efb055dff3b47a0b6515492fcfdc8d0 2023-04-24 18:53:44,959 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 8efb055dff3b47a0b6515492fcfdc8d0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:44,959 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:44,959 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362424328; duration=0sec 2023-04-24 18:53:44,961 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:45,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362484997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362485004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362485106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362485110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,137 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/5a1038fcfd2d4834b42ff4ed976e3273 2023-04-24 18:53:45,151 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/c4d66b80845c45b794fb691c3582b06f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c4d66b80845c45b794fb691c3582b06f 2023-04-24 18:53:45,176 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into c4d66b80845c45b794fb691c3582b06f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:45,176 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:45,176 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362424329; duration=0sec 2023-04-24 18:53:45,176 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:45,214 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1456e675e0e941e8884be72e052194b0 2023-04-24 18:53:45,231 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/03db3d6ecd2941c780c188d336a6f983 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983 2023-04-24 18:53:45,240 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983, entries=150, sequenceid=200, filesize=11.8 K 2023-04-24 18:53:45,244 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/5a1038fcfd2d4834b42ff4ed976e3273 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273 2023-04-24 18:53:45,255 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273, entries=150, sequenceid=200, filesize=11.8 K 2023-04-24 18:53:45,257 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1456e675e0e941e8884be72e052194b0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0 2023-04-24 18:53:45,274 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0, entries=150, sequenceid=200, filesize=11.8 K 2023-04-24 18:53:45,280 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=152.07 KB/155720 for 84739daff3fefab14f3952d64d6f3630 in 789ms, sequenceid=200, compaction requested=false 2023-04-24 18:53:45,280 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:45,280 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:45,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:45,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ced53c1[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:45,288 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:45,288 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:45,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:45,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:45,301 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:45,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:45,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:45,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:45,301 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:45,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,310 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,317 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362424986, max timestamp=1682362425314Num uniques 1104; 2023-04-24 18:53:45,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 18:53:45,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362424986, max timestamp=1682362425314Num uniques 1204; 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:45,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:45,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=76.04 KB heapSize=210.19 KB 2023-04-24 18:53:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:45,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:45,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:45,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362424986, max timestamp=1682362425314Num uniques 1104; 2023-04-24 18:53:45,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 18:53:45,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:45,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:45,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362425347, max timestamp=1682362425359Num uniques 859; 2023-04-24 18:53:45,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362425347, max timestamp=1682362425359Num uniques 859; 2023-04-24 18:53:45,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:45,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362425347, max timestamp=1682362425359Num uniques 859; 2023-04-24 18:53:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:45,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/672f65468f064027a949391a64537a14 2023-04-24 18:53:45,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362485408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362485405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7683, totalHeapSize=17712, min timestamp=1682362425397, max timestamp=1682362425413Num uniques 1068; 2023-04-24 18:53:45,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=7683, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:53:45,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362425397, max timestamp=1682362425413Num uniques 1068; 2023-04-24 18:53:45,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:53:45,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362425397, max timestamp=1682362425413Num uniques 1068; 2023-04-24 18:53:45,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:53:45,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:45,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:45,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d0aafe430e824669baf25c4092bda7cc 2023-04-24 18:53:45,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:45,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:45,440 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:45,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:45,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:45,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:45,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:45,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:45,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:45,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,455 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:45,456 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:45,457 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,457 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:45,457 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:45,457 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:45,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:45,461 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,466 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,469 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,469 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@657f3d78[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:45,473 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:45,473 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,477 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:45,477 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,477 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:45,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:45,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:45,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:45,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:45,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:45,488 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:45,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:45,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:45,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:45,488 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:45,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362425469, max timestamp=1682362425488Num uniques 1027; 2023-04-24 18:53:45,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:45,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362425469, max timestamp=1682362425488Num uniques 1027; 2023-04-24 18:53:45,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:45,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362425469, max timestamp=1682362425488Num uniques 1027; 2023-04-24 18:53:45,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:53:45,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,505 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:45,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:45,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:45,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:45,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362425519, max timestamp=1682362425563Num uniques 1086; 2023-04-24 18:53:45,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362425519, max timestamp=1682362425563Num uniques 1086; 2023-04-24 18:53:45,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362425519, max timestamp=1682362425563Num uniques 1086; 2023-04-24 18:53:45,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:53:45,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:53:45,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:53:45,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362485602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362485603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362485603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362485609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362485611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:45,618 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:45,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:45,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:45,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:45,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:45,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:45,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:45,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:45,638 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,638 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:45,639 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:45,639 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:45,641 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:45,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e886e48[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:45,646 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:45,646 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,647 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:45,647 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:45,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,647 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:45,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:45,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:45,654 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:45,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:45,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:45,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:45,654 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:45,656 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,656 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:45,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:45,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:45,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,667 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:45,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362485713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362485713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362485714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362485719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362485719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,767 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:45,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:45,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:45,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:45,772 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:45,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:45,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:45,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:45,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:45,783 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,783 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:45,783 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:45,783 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555d0789[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:45,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:45,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:45,791 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,791 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:45,793 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:45,801 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:45,802 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:45,803 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,803 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,804 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:45,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:45,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:45,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:45,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:45,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:45,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:45,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:45,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:45,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/92028617ecf241c1b91b489faeee7ebd 2023-04-24 18:53:45,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/672f65468f064027a949391a64537a14 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14 2023-04-24 18:53:45,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14, entries=300, sequenceid=312, filesize=19.0 K 2023-04-24 18:53:45,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d0aafe430e824669baf25c4092bda7cc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc 2023-04-24 18:53:45,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc, entries=150, sequenceid=312, filesize=11.9 K 2023-04-24 18:53:45,915 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:45,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:45,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/92028617ecf241c1b91b489faeee7ebd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd 2023-04-24 18:53:45,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:45,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:45,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:45,930 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:45,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:45,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:45,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362485918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362485926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362485926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362485927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362485927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd, entries=150, sequenceid=312, filesize=11.9 K 2023-04-24 18:53:45,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:45,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:45,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.50 KB/109056, currentSize=147.60 KB/151140 for 84739daff3fefab14f3952d64d6f3630 in 626ms, sequenceid=312, compaction requested=true 2023-04-24 18:53:45,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:45,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:45,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:45,945 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:45,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:45,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:53:45,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:45,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:45,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:45,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:45,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:45,954 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:45,955 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:45,955 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:45,955 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,955 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2f5fa87b883a4c69bb30d27b9230493a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=42.8 K 2023-04-24 18:53:45,956 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,956 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:45,956 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=147.60 KB heapSize=408.23 KB 2023-04-24 18:53:45,956 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:45,956 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:45,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:45,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:45,959 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2f5fa87b883a4c69bb30d27b9230493a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:45,966 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 03db3d6ecd2941c780c188d336a6f983, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682362424051 2023-04-24 18:53:45,967 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 672f65468f064027a949391a64537a14, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424986 2023-04-24 18:53:45,968 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:45,968 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:45,968 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:45,969 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8efb055dff3b47a0b6515492fcfdc8d0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=35.6 K 2023-04-24 18:53:45,970 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8efb055dff3b47a0b6515492fcfdc8d0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:45,971 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5a1038fcfd2d4834b42ff4ed976e3273, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682362424051 2023-04-24 18:53:45,972 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d0aafe430e824669baf25c4092bda7cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424993 2023-04-24 18:53:45,991 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:46,006 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#29 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:46,067 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/0d953ab5e4404ad38cc8c78c0034f6ff as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0d953ab5e4404ad38cc8c78c0034f6ff 2023-04-24 18:53:46,067 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/84367fcc7bd64500ad7103f07d38335e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/84367fcc7bd64500ad7103f07d38335e 2023-04-24 18:53:46,079 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 0d953ab5e4404ad38cc8c78c0034f6ff(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:46,079 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:46,079 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362425945; duration=0sec 2023-04-24 18:53:46,079 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:46,079 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:46,081 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:46,082 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:46,082 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:46,082 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c4d66b80845c45b794fb691c3582b06f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=35.6 K 2023-04-24 18:53:46,082 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c4d66b80845c45b794fb691c3582b06f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682362423585 2023-04-24 18:53:46,083 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 84367fcc7bd64500ad7103f07d38335e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:46,085 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:46,085 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362425945; duration=0sec 2023-04-24 18:53:46,085 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:46,083 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1456e675e0e941e8884be72e052194b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682362424051 2023-04-24 18:53:46,088 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 92028617ecf241c1b91b489faeee7ebd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424993 2023-04-24 18:53:46,104 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#30 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:46,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362486250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362486250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362486254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362486254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362486257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362486359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362486360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362486360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362486360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362486361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,449 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/769dc9377be6411e96657fe3fc5e43c8 2023-04-24 18:53:46,480 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/2eb17ddd41114345961dc51c45145036 2023-04-24 18:53:46,529 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/5dd4f20b5f9f4d9dbce75071acb26882 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/5dd4f20b5f9f4d9dbce75071acb26882 2023-04-24 18:53:46,540 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 5dd4f20b5f9f4d9dbce75071acb26882(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:46,540 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:46,540 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362425945; duration=0sec 2023-04-24 18:53:46,540 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:46,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362486563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362486565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362486565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362486566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362486566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362486867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362486868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362486869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362486870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362486871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,901 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/bf6ff3e554b5471c990df757bfe25733 2023-04-24 18:53:46,909 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/769dc9377be6411e96657fe3fc5e43c8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8 2023-04-24 18:53:46,916 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8, entries=150, sequenceid=399, filesize=11.9 K 2023-04-24 18:53:46,918 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/2eb17ddd41114345961dc51c45145036 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036 2023-04-24 18:53:46,925 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036, entries=150, sequenceid=399, filesize=11.9 K 2023-04-24 18:53:46,929 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/bf6ff3e554b5471c990df757bfe25733 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733 2023-04-24 18:53:46,936 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733, entries=150, sequenceid=399, filesize=11.9 K 2023-04-24 18:53:46,937 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~408.19 KB/417984, currentSize=40.25 KB/41220 for 84739daff3fefab14f3952d64d6f3630 in 981ms, sequenceid=399, compaction requested=false 2023-04-24 18:53:46,937 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:46,937 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:46,937 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:46,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:46,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:46,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@136825b8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:46,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:46,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:46,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:46,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:46,952 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:46,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:46,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:46,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:46,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:46,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:46,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:46,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:46,963 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,968 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,968 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:46,968 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:46,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:46,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:46,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:46,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:46,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:46,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:46,970 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:46,970 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:47,070 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:47,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:47,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:47,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:47,077 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:47,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:47,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:47,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,080 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:47,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:47,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:47,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:47,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:47,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:47,083 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:47,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:47,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:47,084 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,084 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:47,084 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:47,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:47,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362427389, max timestamp=1682362427395Num uniques 859; 2023-04-24 18:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362427389, max timestamp=1682362427395Num uniques 859; 2023-04-24 18:53:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:47,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362427389, max timestamp=1682362427395Num uniques 859; 2023-04-24 18:53:47,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362487411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362487412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362487415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362487415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362487421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,510 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/2cd909b6dae94dd9bb6caead0d4132d5 2023-04-24 18:53:47,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362487523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362487523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362487523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362487524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362487527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,546 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/feabec8648894320b99196fa79643951 2023-04-24 18:53:47,598 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/62b6111110374ef1b6fe6e5d678090fd 2023-04-24 18:53:47,614 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/2cd909b6dae94dd9bb6caead0d4132d5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5 2023-04-24 18:53:47,623 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5, entries=150, sequenceid=411, filesize=11.9 K 2023-04-24 18:53:47,624 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/feabec8648894320b99196fa79643951 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951 2023-04-24 18:53:47,640 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951, entries=150, sequenceid=411, filesize=11.9 K 2023-04-24 18:53:47,642 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/62b6111110374ef1b6fe6e5d678090fd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd 2023-04-24 18:53:47,651 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd, entries=150, sequenceid=411, filesize=11.9 K 2023-04-24 18:53:47,652 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 84739daff3fefab14f3952d64d6f3630 in 568ms, sequenceid=411, compaction requested=true 2023-04-24 18:53:47,652 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:47,652 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,652 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fe04e50[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:47,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:47,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:47,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:47,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:47,676 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:47,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:47,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:47,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:47,677 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:47,679 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,679 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:47,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,679 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,691 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:47,691 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:47,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:47,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:47,691 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:47,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:47,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,693 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:47,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-24 18:53:47,730 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:47,730 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:53:47,730 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:47,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:53:47,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:47,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:53:47,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362487734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362487735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362487735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362487740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362487741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,757 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/0e5dc240368543fcbce11d34190e0248 2023-04-24 18:53:47,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:47,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:47,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:47,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:47,800 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:47,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:47,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:47,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,804 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:47,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:47,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:47,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:47,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:47,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:47,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:47,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:47,816 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f79bca[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:47,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:47,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:47,824 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,824 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:47,826 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:47,834 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:47,834 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:47,834 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:47,834 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:47,834 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:47,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:47,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:47,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:47,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:47,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:47,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:47,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:47,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:47,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:47,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362487848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362487848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362487850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362487852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362487852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,945 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:47,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:47,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:47,951 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:47,951 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:47,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:47,953 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,953 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:47,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:47,963 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,963 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:47,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:47,970 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:47,970 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:47,974 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f2ba952[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:47,977 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:47,977 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:47,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:47,981 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:47,988 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:47,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:47,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:47,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:47,988 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:47,990 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,991 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,991 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:47,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:47,992 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:47,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:47,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:47,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:47,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:48,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:48,000 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:48,000 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:48,001 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,001 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362488051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362488051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362488056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362488056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362488056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,100 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:48,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:48,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:48,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:48,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:48,106 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:48,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,108 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:48,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:48,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,111 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:48,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:48,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:48,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:48,118 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3c02ea[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:48,127 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:48,127 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,129 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:48,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:48,134 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:48,141 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:48,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:48,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:48,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:48,142 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:48,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,157 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,157 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,157 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:48,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:48,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:48,168 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:48,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/e9ce963f1ed844908ad5d7808f82e365 2023-04-24 18:53:48,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:48,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:48,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:48,273 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:48,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:48,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:48,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,277 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:48,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:48,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:48,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:48,283 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,283 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:48,283 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:48,283 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a35fd3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:48,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:48,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:48,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:48,298 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:48,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:48,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:48,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:48,298 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:48,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:48,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:48,309 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362488354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362488354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362488362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362488361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362488363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:48,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:48,428 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:48,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:48,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:48,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,432 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,435 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:48,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:48,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:48,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:48,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:48,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,439 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:48,440 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,440 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:48,440 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:48,441 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:48,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed4bdc6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:48,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:48,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,450 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:48,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:48,451 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,451 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:48,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:48,457 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:48,457 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:48,457 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:48,457 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:48,457 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:48,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,482 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:48,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:48,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:48,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:48,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:48,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,593 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:48,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:48,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:48,597 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:48,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:48,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:48,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/0ec491c244a942459ba1d838d4d9fe0f 2023-04-24 18:53:48,601 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:48,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:48,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:48,604 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,604 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:48,604 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:48,604 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:48,606 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:48,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/0e5dc240368543fcbce11d34190e0248 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248 2023-04-24 18:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb7ee77[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:48,612 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:48,612 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,613 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:48,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:48,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248, entries=200, sequenceid=449, filesize=14.3 K 2023-04-24 18:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/e9ce963f1ed844908ad5d7808f82e365 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365 2023-04-24 18:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:48,621 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:48,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:48,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:48,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:48,621 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:48,623 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,623 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,623 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:48,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,624 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365, entries=150, sequenceid=449, filesize=11.9 K 2023-04-24 18:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/0ec491c244a942459ba1d838d4d9fe0f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f 2023-04-24 18:53:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:48,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:48,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:48,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:48,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:48,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,651 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:48,651 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:48,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f, entries=150, sequenceid=449, filesize=11.9 K 2023-04-24 18:53:48,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 84739daff3fefab14f3952d64d6f3630 in 923ms, sequenceid=449, compaction requested=true 2023-04-24 18:53:48,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:48,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:48,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:48,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:53:48,653 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:48,653 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:48,656 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49052 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:48,657 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:48,657 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:48,657 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,657 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:48,657 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,657 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/84367fcc7bd64500ad7103f07d38335e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=47.9 K 2023-04-24 18:53:48,657 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0d953ab5e4404ad38cc8c78c0034f6ff, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=50.3 K 2023-04-24 18:53:48,658 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 84367fcc7bd64500ad7103f07d38335e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424993 2023-04-24 18:53:48,660 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0d953ab5e4404ad38cc8c78c0034f6ff, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424993 2023-04-24 18:53:48,661 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2eb17ddd41114345961dc51c45145036, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682362425601 2023-04-24 18:53:48,662 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting feabec8648894320b99196fa79643951, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682362426250 2023-04-24 18:53:48,662 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 769dc9377be6411e96657fe3fc5e43c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682362425601 2023-04-24 18:53:48,663 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e9ce963f1ed844908ad5d7808f82e365, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:48,663 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2cd909b6dae94dd9bb6caead0d4132d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682362426250 2023-04-24 18:53:48,665 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0e5dc240368543fcbce11d34190e0248, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:48,679 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:48,682 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:48,751 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:48,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:48,759 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:48,759 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,762 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,762 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:48,762 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,762 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:48,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:48,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:48,765 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:48,765 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:48,765 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:48,765 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:48,766 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:48,766 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:48,766 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:48,766 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:48,766 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:48,782 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/6c1d607e33c34e39945154b3665e10c2 2023-04-24 18:53:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:48,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:48,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362428877, max timestamp=1682362428883Num uniques 859; 2023-04-24 18:53:48,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:48,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362428877, max timestamp=1682362428883Num uniques 859; 2023-04-24 18:53:48,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:48,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362428877, max timestamp=1682362428883Num uniques 859; 2023-04-24 18:53:48,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:48,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362488902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362488903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:48,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362428899, max timestamp=1682362428905Num uniques 1068; 2023-04-24 18:53:48,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362488905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:48,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 18:53:48,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362428899, max timestamp=1682362428905Num uniques 1068; 2023-04-24 18:53:48,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362428899, max timestamp=1682362428905Num uniques 1068; 2023-04-24 18:53:48,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 18:53:48,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 18:53:48,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362428931, max timestamp=1682362428944Num uniques 1077; 2023-04-24 18:53:48,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:53:48,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362428931, max timestamp=1682362428944Num uniques 1077; 2023-04-24 18:53:48,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:53:48,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362428931, max timestamp=1682362428944Num uniques 1077; 2023-04-24 18:53:48,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:53:48,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362428965, max timestamp=1682362428996Num uniques 1086; 2023-04-24 18:53:48,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362428965, max timestamp=1682362428996Num uniques 1086; 2023-04-24 18:53:48,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:53:48,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:53:49,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362428965, max timestamp=1682362428996Num uniques 1086; 2023-04-24 18:53:49,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362429013, max timestamp=1682362429027Num uniques 1045; 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362429013, max timestamp=1682362429027Num uniques 1045; 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362429013, max timestamp=1682362429027Num uniques 1045; 2023-04-24 18:53:49,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:53:49,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362489041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362489042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362489043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362489044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362489045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,099 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/7e62f5982c4143d8bcead9ae718d04de as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7e62f5982c4143d8bcead9ae718d04de 2023-04-24 18:53:49,102 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/2247842d7dbc4f8481ebf88e4c1bfb9a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2247842d7dbc4f8481ebf88e4c1bfb9a 2023-04-24 18:53:49,108 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 7e62f5982c4143d8bcead9ae718d04de(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:49,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:49,108 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=12, startTime=1682362428653; duration=0sec 2023-04-24 18:53:49,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:49,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:49,110 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 2247842d7dbc4f8481ebf88e4c1bfb9a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:49,110 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:49,110 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=12, startTime=1682362428653; duration=0sec 2023-04-24 18:53:49,110 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:49,110 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49052 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:49,110 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:49,111 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:49,111 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/5dd4f20b5f9f4d9dbce75071acb26882, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=47.9 K 2023-04-24 18:53:49,111 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5dd4f20b5f9f4d9dbce75071acb26882, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1682362424993 2023-04-24 18:53:49,112 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting bf6ff3e554b5471c990df757bfe25733, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682362425601 2023-04-24 18:53:49,112 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 62b6111110374ef1b6fe6e5d678090fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682362426250 2023-04-24 18:53:49,113 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0ec491c244a942459ba1d838d4d9fe0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:49,123 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:49,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362489148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362489149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35514 deadline: 1682362489153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35444 deadline: 1682362489154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362489154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,156 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/f09eb8c47fec493299d96c3f6b4b29f3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f09eb8c47fec493299d96c3f6b4b29f3 2023-04-24 18:53:49,166 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into f09eb8c47fec493299d96c3f6b4b29f3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:49,167 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:49,167 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=12, startTime=1682362428653; duration=0sec 2023-04-24 18:53:49,167 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:49,222 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/5484a803c2234d8189eb7bde744ea41d 2023-04-24 18:53:49,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362489354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362489355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362489358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362489358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362489358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362489657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362489660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,663 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/33d09dc51b0943ac82c23d64101d8f07 2023-04-24 18:53:49,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362489664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362489665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:49,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362489665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/6c1d607e33c34e39945154b3665e10c2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2 2023-04-24 18:53:49,680 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2, entries=150, sequenceid=457, filesize=11.9 K 2023-04-24 18:53:49,681 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/5484a803c2234d8189eb7bde744ea41d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d 2023-04-24 18:53:49,687 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d, entries=150, sequenceid=457, filesize=11.9 K 2023-04-24 18:53:49,689 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/33d09dc51b0943ac82c23d64101d8f07 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07 2023-04-24 18:53:49,695 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07, entries=150, sequenceid=457, filesize=11.9 K 2023-04-24 18:53:49,696 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 84739daff3fefab14f3952d64d6f3630 in 931ms, sequenceid=457, compaction requested=false 2023-04-24 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:49,696 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:49,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31719c59[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:49,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:49,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,708 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:49,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:49,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:49,709 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,709 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:49,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:49,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:49,715 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:49,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:49,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:49,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:49,715 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:49,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:49,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,732 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:49,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:49,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:49,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:49,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:49,738 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:49,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:49,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:49,838 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:49,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:49,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:49,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:49,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:49,843 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:49,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:49,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:49,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:49,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:49,847 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,847 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:49,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:49,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:49,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:49,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:49,851 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:49,851 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:49,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:49,852 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:49,852 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:49,853 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=154.31 KB heapSize=417.94 KB 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:49,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:49,875 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/98bcedad954e4240b579a6106af3229a 2023-04-24 18:53:49,907 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/6ea3fc3b6eb14ec493af7ee7def13993 2023-04-24 18:53:49,926 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7d021c2232a54e839b2c013e90fde8bd 2023-04-24 18:53:49,933 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/98bcedad954e4240b579a6106af3229a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a 2023-04-24 18:53:49,939 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a, entries=150, sequenceid=563, filesize=11.9 K 2023-04-24 18:53:49,944 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/6ea3fc3b6eb14ec493af7ee7def13993 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993 2023-04-24 18:53:49,962 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993, entries=150, sequenceid=563, filesize=11.9 K 2023-04-24 18:53:49,963 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7d021c2232a54e839b2c013e90fde8bd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd 2023-04-24 18:53:49,971 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd, entries=150, sequenceid=563, filesize=11.9 K 2023-04-24 18:53:49,972 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~417.89 KB/427920, currentSize=0 B/0 for 84739daff3fefab14f3952d64d6f3630 in 120ms, sequenceid=563, compaction requested=true 2023-04-24 18:53:49,972 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:49,972 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:49,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:49,980 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b361063[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:49,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:49,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:49,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:49,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:49,989 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:49,994 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:49,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:49,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:49,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:49,994 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:49,996 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:49,996 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,996 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:49,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:49,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:49,997 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:50,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:50,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,016 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:50,016 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:50,116 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:50,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:50,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:50,121 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:50,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,126 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:50,126 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:50,127 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,128 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,129 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:50,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:50,134 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,134 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:50,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f7b24e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:50,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:50,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:50,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:50,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:50,164 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:50,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:50,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:50,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:50,164 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:50,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,173 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,173 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:50,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:50,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,177 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:50,177 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:50,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:50,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:50,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:53:50,180 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:50,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:50,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:50,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:50,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:50,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:50,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682362430197, max timestamp=1682362430213Num uniques 859; 2023-04-24 18:53:50,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:53:50,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682362430197, max timestamp=1682362430213Num uniques 859; 2023-04-24 18:53:50,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:53:50,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362430197, max timestamp=1682362430213Num uniques 859; 2023-04-24 18:53:50,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:53:50,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/08eeb2fdf56e4df1aa4d66e0e394eea2 2023-04-24 18:53:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362490237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362490238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362490239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362490241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362490242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,277 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:50,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:50,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:50,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:50,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:50,285 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:50,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:50,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:50,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,325 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,325 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:50,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:50,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:50,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:50,327 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,327 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:50,327 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:50,327 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:50,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362490346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362490347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362490347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362490349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362490350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,355 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d02a71[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:50,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:50,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:50,393 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,393 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:50,462 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:50,466 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:50,466 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:50,466 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:50,466 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:50,466 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:50,516 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362490549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362490549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362490551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362490554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362490556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:50,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:50,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:50,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:50,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/8c2d1baa6ebd49008a375693f056779f 2023-04-24 18:53:50,729 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:50,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:50,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:50,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:50,733 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:50,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:50,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:50,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,770 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:50,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:50,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,816 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7efb06[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:50,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:50,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:50,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:50,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362490855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35508 deadline: 1682362490856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362490857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362490857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362490859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:50,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:50,911 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:50,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:50,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:50,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:50,911 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:50,977 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,977 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,977 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:50,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:50,978 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,980 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,039 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,039 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:51,039 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:51,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:51,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:51,039 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:51,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:51,039 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:51,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:51,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/c8d08ed0b27045e481908feb188e659a 2023-04-24 18:53:51,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/08eeb2fdf56e4df1aa4d66e0e394eea2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2 2023-04-24 18:53:51,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2, entries=200, sequenceid=575, filesize=14.3 K 2023-04-24 18:53:51,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/8c2d1baa6ebd49008a375693f056779f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f 2023-04-24 18:53:51,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f, entries=150, sequenceid=575, filesize=11.9 K 2023-04-24 18:53:51,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/c8d08ed0b27045e481908feb188e659a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a 2023-04-24 18:53:51,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a, entries=150, sequenceid=575, filesize=11.9 K 2023-04-24 18:53:51,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 84739daff3fefab14f3952d64d6f3630 in 935ms, sequenceid=575, compaction requested=true 2023-04-24 18:53:51,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:51,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:51,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:51,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:51,115 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:51,115 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:51,117 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51628 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:51,117 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:51,118 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:51,118 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:51,118 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2247842d7dbc4f8481ebf88e4c1bfb9a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=48.0 K 2023-04-24 18:53:51,118 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:51,118 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:51,118 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7e62f5982c4143d8bcead9ae718d04de, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=50.4 K 2023-04-24 18:53:51,118 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2247842d7dbc4f8481ebf88e4c1bfb9a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:51,120 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7e62f5982c4143d8bcead9ae718d04de, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:51,121 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5484a803c2234d8189eb7bde744ea41d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682362427732 2023-04-24 18:53:51,121 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6c1d607e33c34e39945154b3665e10c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682362427732 2023-04-24 18:53:51,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6ea3fc3b6eb14ec493af7ee7def13993, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682362429038 2023-04-24 18:53:51,122 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 98bcedad954e4240b579a6106af3229a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682362429038 2023-04-24 18:53:51,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8c2d1baa6ebd49008a375693f056779f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:51,122 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 08eeb2fdf56e4df1aa4d66e0e394eea2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:51,134 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#51 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:51,137 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:51,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:51,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:51,147 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:51,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:51,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:51,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:51,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,162 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/6c656866a6954b5f995a029bea8a9a5a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c656866a6954b5f995a029bea8a9a5a 2023-04-24 18:53:51,170 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 6c656866a6954b5f995a029bea8a9a5a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:51,170 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:51,170 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=12, startTime=1682362431115; duration=0sec 2023-04-24 18:53:51,170 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:51,170 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:53:51,173 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:53:51,173 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:51,173 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:51,174 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f09eb8c47fec493299d96c3f6b4b29f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=48.0 K 2023-04-24 18:53:51,174 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f09eb8c47fec493299d96c3f6b4b29f3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682362427410 2023-04-24 18:53:51,175 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 33d09dc51b0943ac82c23d64101d8f07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682362427732 2023-04-24 18:53:51,176 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7d021c2232a54e839b2c013e90fde8bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1682362429038 2023-04-24 18:53:51,176 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c8d08ed0b27045e481908feb188e659a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:51,185 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:51,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:51,187 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:51,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:51,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:51,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:51,209 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:51,209 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:51,210 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:51,214 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:51,246 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/375e6d01bfcd4092ab50e1772ceb747d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/375e6d01bfcd4092ab50e1772ceb747d 2023-04-24 18:53:51,254 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/1a4076d6ecfe4f47ad93221d0475d073 2023-04-24 18:53:51,258 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 375e6d01bfcd4092ab50e1772ceb747d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:51,258 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:51,258 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=12, startTime=1682362431115; duration=0sec 2023-04-24 18:53:51,258 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:51,287 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/a494e3d8305545a69c004fd11af96222 2023-04-24 18:53:51,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:51,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:51,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362491370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362491371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362491371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362491372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362491375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362491477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362491479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362491480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362491482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362491483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,565 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/c0c06acf136b4910a331d75eee2dc24d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/c0c06acf136b4910a331d75eee2dc24d 2023-04-24 18:53:51,574 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into c0c06acf136b4910a331d75eee2dc24d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:51,575 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:51,575 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=12, startTime=1682362431115; duration=0sec 2023-04-24 18:53:51,575 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:51,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362491683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362491684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362491685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362491685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:51,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362491691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,727 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3f2ca82cc2a841a287c77e83bcb197c4 2023-04-24 18:53:51,734 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/1a4076d6ecfe4f47ad93221d0475d073 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073 2023-04-24 18:53:51,742 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073, entries=150, sequenceid=610, filesize=11.9 K 2023-04-24 18:53:51,743 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/a494e3d8305545a69c004fd11af96222 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222 2023-04-24 18:53:51,750 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222, entries=150, sequenceid=610, filesize=11.9 K 2023-04-24 18:53:51,752 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3f2ca82cc2a841a287c77e83bcb197c4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4 2023-04-24 18:53:51,766 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4, entries=150, sequenceid=610, filesize=11.9 K 2023-04-24 18:53:51,767 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 84739daff3fefab14f3952d64d6f3630 in 558ms, sequenceid=610, compaction requested=false 2023-04-24 18:53:51,767 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:51,767 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:51,767 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:51,793 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d057a2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:51,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:51,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,816 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:51,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:51,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:51,842 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:51,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:51,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:51,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:51,843 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:51,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:51,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:51,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,874 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,893 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:51,893 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:51,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:51,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:51,895 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,895 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:51,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:51,895 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:51,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:51,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:51,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:51,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:51,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:51,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:51,896 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:51,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:51,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:51,996 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:51,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:52,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:52,002 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:52,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:52,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:52,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,004 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:52,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:52,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:52,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:52,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:52,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7858e294[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:52,009 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:52,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:52,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:52,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:52,014 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:52,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:52,016 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:52,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:52,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:52,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:52,016 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:52,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,018 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:52,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:52,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,039 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:52,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:52,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:52,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,143 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:52,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:52,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:52,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:52,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:52,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:52,148 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:52,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:52,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae9cc4c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:52,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:52,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:52,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:52,162 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:52,162 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,162 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:52,162 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:52,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:52,166 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:52,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:52,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:52,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:52,167 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:52,169 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,169 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,169 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,173 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:52,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:52,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:52,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,177 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:52,177 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:52,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:52,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:52,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:52,284 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:52,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:52,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:52,288 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,288 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:52,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:52,292 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,292 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:52,292 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:52,292 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:52,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,296 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3336a8dc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:52,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:52,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:52,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:52,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:52,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:52,303 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:52,305 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:52,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:52,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:52,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:52,306 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:52,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,308 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,312 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,312 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,312 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,312 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:52,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:52,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,312 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:52,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:52,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:52,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/8a94782333f84c47bb67e36e17b9004c 2023-04-24 18:53:52,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:52,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:52,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:52,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:52,425 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:52,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:52,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:52,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,429 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:52,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:52,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:52,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:52,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:52,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d3bc2e96a9194771b2e97e5fa6db64fc 2023-04-24 18:53:52,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244c845f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:52,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:52,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,455 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:52,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:52,458 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,459 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:52,467 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:52,467 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:52,467 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:52,467 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:52,467 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:52,470 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,470 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:52,470 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:52,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,472 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:52,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7e75b6454022431bbf4128466bbc9467 2023-04-24 18:53:52,474 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,479 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,479 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:52,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:52,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:52,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:52,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:52,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:52,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:52,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:52,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:52,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/8a94782333f84c47bb67e36e17b9004c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c 2023-04-24 18:53:52,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c, entries=150, sequenceid=626, filesize=11.9 K 2023-04-24 18:53:52,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d3bc2e96a9194771b2e97e5fa6db64fc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc 2023-04-24 18:53:52,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc, entries=150, sequenceid=626, filesize=11.9 K 2023-04-24 18:53:52,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/7e75b6454022431bbf4128466bbc9467 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467 2023-04-24 18:53:52,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467, entries=150, sequenceid=626, filesize=11.9 K 2023-04-24 18:53:52,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 84739daff3fefab14f3952d64d6f3630 in 537ms, sequenceid=626, compaction requested=true 2023-04-24 18:53:52,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:52,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:52,528 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:52,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:52,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:53:52,529 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:52,530 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:52,530 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:52,530 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,530 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c656866a6954b5f995a029bea8a9a5a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.3 K 2023-04-24 18:53:52,530 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:52,530 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:52,531 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,531 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6c656866a6954b5f995a029bea8a9a5a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:52,531 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/c0c06acf136b4910a331d75eee2dc24d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.3 K 2023-04-24 18:53:52,531 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1a4076d6ecfe4f47ad93221d0475d073, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682362430235 2023-04-24 18:53:52,531 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c0c06acf136b4910a331d75eee2dc24d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:52,532 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8a94782333f84c47bb67e36e17b9004c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:52,532 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a494e3d8305545a69c004fd11af96222, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682362430235 2023-04-24 18:53:52,533 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d3bc2e96a9194771b2e97e5fa6db64fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:52,545 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:52,553 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#61 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:52,581 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:52,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:52,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:52,585 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:52,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:52,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:52,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,587 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:52,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:52,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:52,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:52,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:52,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:52,600 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:52,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:52,601 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:52,601 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:52,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:52,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,975 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/0aa55e500d6e4afca856241e090782e9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/0aa55e500d6e4afca856241e090782e9 2023-04-24 18:53:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362492975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362492976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362492978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362492980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362492980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:52,985 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/750ef5b8168f43a4b4e5a2a091fdd013 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/750ef5b8168f43a4b4e5a2a091fdd013 2023-04-24 18:53:52,994 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 0aa55e500d6e4afca856241e090782e9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:52,994 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:52,994 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362432528; duration=0sec 2023-04-24 18:53:52,994 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:52,994 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:52,994 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 750ef5b8168f43a4b4e5a2a091fdd013(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:52,994 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:52,994 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362432528; duration=0sec 2023-04-24 18:53:52,994 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:52,996 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:52,996 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:52,996 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:52,996 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/375e6d01bfcd4092ab50e1772ceb747d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.3 K 2023-04-24 18:53:52,997 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 375e6d01bfcd4092ab50e1772ceb747d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682362430168 2023-04-24 18:53:52,998 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3f2ca82cc2a841a287c77e83bcb197c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682362430235 2023-04-24 18:53:52,998 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7e75b6454022431bbf4128466bbc9467, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:53,009 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:53,032 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/95e447c3d89d4906984183dcfe36e772 2023-04-24 18:53:53,048 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/4e099128c5904e99b2d677c73233e118 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4e099128c5904e99b2d677c73233e118 2023-04-24 18:53:53,065 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 4e099128c5904e99b2d677c73233e118(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:53,065 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:53,065 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362432528; duration=0sec 2023-04-24 18:53:53,065 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:53,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362493279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362493281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362493281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362493284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362493284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,472 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/4457cb43e8f94a8295c693644d2ad5e7 2023-04-24 18:53:53,495 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3b12bb5ab39f44fd9b27fb4db519a8b4 2023-04-24 18:53:53,502 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/95e447c3d89d4906984183dcfe36e772 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772 2023-04-24 18:53:53,509 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772, entries=150, sequenceid=644, filesize=11.9 K 2023-04-24 18:53:53,510 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/4457cb43e8f94a8295c693644d2ad5e7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7 2023-04-24 18:53:53,517 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7, entries=150, sequenceid=644, filesize=11.9 K 2023-04-24 18:53:53,518 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/3b12bb5ab39f44fd9b27fb4db519a8b4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4 2023-04-24 18:53:53,525 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4, entries=150, sequenceid=644, filesize=11.9 K 2023-04-24 18:53:53,526 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 84739daff3fefab14f3952d64d6f3630 in 926ms, sequenceid=644, compaction requested=false 2023-04-24 18:53:53,526 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:53,526 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:53,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbf8e00[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:53,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:53,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:53,539 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,539 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:53,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:53,546 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:53,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:53,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:53,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:53,546 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:53,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,555 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:53,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:53,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:53,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:53,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:53,559 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:53,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:53,560 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:53,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:53,660 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:53,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:53,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:53,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:53,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:53,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:53,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:53,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:53,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:53,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:53,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:53,670 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:53,670 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:53,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:53,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:53,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362493795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362493795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362493796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362493797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362493799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362493901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362493901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362493902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362493902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362493902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,080 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/9d7d5f7bafe1465f9e04c1e9f459c86a 2023-04-24 18:53:54,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362494105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362494105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362494105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362494105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362494106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362494409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362494410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362494411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362494412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362494412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,503 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/785bcd351c094bc7bea099f74e7fbefa 2023-04-24 18:53:54,517 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/0abb0187481948e8bacecf7b48f746ac 2023-04-24 18:53:54,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/9d7d5f7bafe1465f9e04c1e9f459c86a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a 2023-04-24 18:53:54,530 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a, entries=150, sequenceid=661, filesize=11.9 K 2023-04-24 18:53:54,531 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/785bcd351c094bc7bea099f74e7fbefa as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa 2023-04-24 18:53:54,540 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa, entries=150, sequenceid=661, filesize=11.9 K 2023-04-24 18:53:54,541 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/0abb0187481948e8bacecf7b48f746ac as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac 2023-04-24 18:53:54,561 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac, entries=150, sequenceid=661, filesize=11.9 K 2023-04-24 18:53:54,562 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 84739daff3fefab14f3952d64d6f3630 in 892ms, sequenceid=661, compaction requested=true 2023-04-24 18:53:54,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:54,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:54,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:54,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:54,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f30ba97[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:54,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:54,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:54,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:54,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:54,576 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:54,576 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:54,576 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:54,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:54,577 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:54,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:54,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,582 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,589 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,589 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:54,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:54,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:54,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:54,590 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:54,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:54,590 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:54,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:54,690 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:54,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:54,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:54,694 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:54,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:54,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:54,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:54,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:54,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:54,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:54,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:54,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:54,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:54,702 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:54,702 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:54,702 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 18:53:54,702 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:54,703 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:54,703 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:54,703 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:54,703 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:54,704 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:54,729 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/d8d695d56d2f4f02b5dabe35e17ad6d7 2023-04-24 18:53:54,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:54,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:54,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362494924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362494925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362494927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362494928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:54,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362494928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362495030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362495030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362495030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362495032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362495032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,167 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/afed1d9a613d4658870c0863fa0593ca 2023-04-24 18:53:55,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362495233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362495235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362495235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362495237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362495237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362495537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362495537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362495537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362495553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:55,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362495554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,584 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/07a452cf622040aa897c57b9fe93c10c 2023-04-24 18:53:55,593 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/d8d695d56d2f4f02b5dabe35e17ad6d7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7 2023-04-24 18:53:55,599 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7, entries=150, sequenceid=679, filesize=11.9 K 2023-04-24 18:53:55,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/afed1d9a613d4658870c0863fa0593ca as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca 2023-04-24 18:53:55,605 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca, entries=150, sequenceid=679, filesize=11.9 K 2023-04-24 18:53:55,606 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/07a452cf622040aa897c57b9fe93c10c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c 2023-04-24 18:53:55,612 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c, entries=150, sequenceid=679, filesize=11.9 K 2023-04-24 18:53:55,612 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 84739daff3fefab14f3952d64d6f3630 in 910ms, sequenceid=679, compaction requested=true 2023-04-24 18:53:55,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:55,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:55,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:55,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c691e5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:55,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:55,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,620 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:55,620 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,620 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:55,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:55,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:55,625 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:55,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:55,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:55,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:55,625 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:55,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:55,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:55,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:55,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:55,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,629 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:55,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:55,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:55,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:55,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:55,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:55,634 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:55,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:55,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:55,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:55,634 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:55,634 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:55,734 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:55,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:55,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:55,738 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:55,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:55,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:55,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:55,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:55,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:55,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:55,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:55,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:55,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:55,742 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:55,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:55,752 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/893e246f2c904bebb80daea11f0a2fa4 2023-04-24 18:53:55,765 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f42d0d41adf04f8298df1bce6301bc74 2023-04-24 18:53:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:56,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362496057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,181 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/55ff2264cd2c4413b78c250fc14d4625 2023-04-24 18:53:56,190 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/893e246f2c904bebb80daea11f0a2fa4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4 2023-04-24 18:53:56,199 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4, entries=150, sequenceid=694, filesize=11.9 K 2023-04-24 18:53:56,204 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f42d0d41adf04f8298df1bce6301bc74 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74 2023-04-24 18:53:56,219 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74, entries=150, sequenceid=694, filesize=11.9 K 2023-04-24 18:53:56,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/55ff2264cd2c4413b78c250fc14d4625 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625 2023-04-24 18:53:56,225 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625, entries=150, sequenceid=694, filesize=11.9 K 2023-04-24 18:53:56,226 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 84739daff3fefab14f3952d64d6f3630 in 484ms, sequenceid=694, compaction requested=true 2023-04-24 18:53:56,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:56,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,229 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e880fe[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:56,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:56,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:56,242 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:56,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:56,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:56,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:56,242 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:56,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:56,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:56,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,258 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:56,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:56,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:56,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:56,364 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:56,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:56,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:56,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,368 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:56,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:56,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 18:53:56,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:56,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:56,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,373 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:56,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:56,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:56,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:56,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:56,392 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,392 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:56,392 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,392 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,394 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e40ca4d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:56,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:56,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:56,402 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:56,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:56,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:56,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:56,402 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:56,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,405 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/2a83d45527784f4faf9928f40e0a77ae 2023-04-24 18:53:56,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:56,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:56,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/7072a3f2312f467aa154712895dd4383 2023-04-24 18:53:56,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1f963be369634aa2bd8a2e6196ff7100 2023-04-24 18:53:56,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/2a83d45527784f4faf9928f40e0a77ae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae 2023-04-24 18:53:56,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae, entries=150, sequenceid=713, filesize=11.9 K 2023-04-24 18:53:56,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/7072a3f2312f467aa154712895dd4383 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383 2023-04-24 18:53:56,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383, entries=150, sequenceid=713, filesize=11.9 K 2023-04-24 18:53:56,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1f963be369634aa2bd8a2e6196ff7100 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100 2023-04-24 18:53:56,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100, entries=150, sequenceid=713, filesize=11.9 K 2023-04-24 18:53:56,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 84739daff3fefab14f3952d64d6f3630 in 125ms, sequenceid=713, compaction requested=true 2023-04-24 18:53:56,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:56,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:56,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:56,495 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:53:56,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:56,495 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:53:56,499 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:53:56,499 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:56,499 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,499 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:53:56,499 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/0aa55e500d6e4afca856241e090782e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=72.1 K 2023-04-24 18:53:56,499 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:56,500 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,500 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/750ef5b8168f43a4b4e5a2a091fdd013, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=72.1 K 2023-04-24 18:53:56,500 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0aa55e500d6e4afca856241e090782e9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:56,500 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 750ef5b8168f43a4b4e5a2a091fdd013, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:56,500 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4457cb43e8f94a8295c693644d2ad5e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682362432037 2023-04-24 18:53:56,501 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 95e447c3d89d4906984183dcfe36e772, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682362432037 2023-04-24 18:53:56,501 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 785bcd351c094bc7bea099f74e7fbefa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1682362432661 2023-04-24 18:53:56,501 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9d7d5f7bafe1465f9e04c1e9f459c86a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1682362432661 2023-04-24 18:53:56,501 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting afed1d9a613d4658870c0863fa0593ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1682362433791 2023-04-24 18:53:56,501 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d8d695d56d2f4f02b5dabe35e17ad6d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1682362433791 2023-04-24 18:53:56,502 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f42d0d41adf04f8298df1bce6301bc74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682362434923 2023-04-24 18:53:56,502 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 893e246f2c904bebb80daea11f0a2fa4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682362434923 2023-04-24 18:53:56,502 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7072a3f2312f467aa154712895dd4383, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:56,502 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2a83d45527784f4faf9928f40e0a77ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:56,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:56,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:56,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:56,516 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:56,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:56,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:56,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,519 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:56,522 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:56,522 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:56,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:56,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:56,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:56,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:56,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,531 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24be2b10[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/eab576403d8a4214a668869d67f2b424 2023-04-24 18:53:56,537 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:56,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:56,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:56,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:56,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:56,546 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:56,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:56,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:56,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:56,546 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:56,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,554 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,554 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:56,554 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:56,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,555 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,555 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:56,567 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/8b58b4950a474c3db418a2b214b14aec as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8b58b4950a474c3db418a2b214b14aec 2023-04-24 18:53:56,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/28266ca1576348a7a903421ce4b21271 2023-04-24 18:53:56,582 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 8b58b4950a474c3db418a2b214b14aec(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:56,583 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:56,583 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=10, startTime=1682362436495; duration=0sec 2023-04-24 18:53:56,583 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:56,583 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:53:56,586 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:53:56,586 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:56,586 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,586 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4e099128c5904e99b2d677c73233e118, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=72.1 K 2023-04-24 18:53:56,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/f805bd8f77664311a979ca2ab229390c 2023-04-24 18:53:56,586 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4e099128c5904e99b2d677c73233e118, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1682362431371 2023-04-24 18:53:56,587 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3b12bb5ab39f44fd9b27fb4db519a8b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1682362432037 2023-04-24 18:53:56,587 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0abb0187481948e8bacecf7b48f746ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1682362432661 2023-04-24 18:53:56,588 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 07a452cf622040aa897c57b9fe93c10c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1682362433791 2023-04-24 18:53:56,588 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 55ff2264cd2c4413b78c250fc14d4625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682362434923 2023-04-24 18:53:56,589 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1f963be369634aa2bd8a2e6196ff7100, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:56,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/eab576403d8a4214a668869d67f2b424 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424 2023-04-24 18:53:56,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424, entries=150, sequenceid=728, filesize=11.9 K 2023-04-24 18:53:56,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/28266ca1576348a7a903421ce4b21271 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271 2023-04-24 18:53:56,608 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:56,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271, entries=150, sequenceid=728, filesize=11.9 K 2023-04-24 18:53:56,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/f805bd8f77664311a979ca2ab229390c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c 2023-04-24 18:53:56,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c, entries=150, sequenceid=728, filesize=11.9 K 2023-04-24 18:53:56,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 84739daff3fefab14f3952d64d6f3630 in 121ms, sequenceid=728, compaction requested=false 2023-04-24 18:53:56,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:56,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:56,648 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:56,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:56,657 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:56,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:56,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:56,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:56,667 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:56,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:56,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:56,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/e92eb0ca532943fb92c34f1a888f24b9 2023-04-24 18:53:56,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:56,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:56,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:56,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:56,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:56,677 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,679 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,679 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9dadfdb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:56,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:56,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:56,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f24b33d899ea46cca6ccc205e2205f79 2023-04-24 18:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:56,701 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:56,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:56,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:56,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:56,701 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:56,704 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,704 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,704 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,709 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,721 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:56,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:56,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:56,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:56,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:56,722 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,722 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:56,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:56,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:56,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:56,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:56,827 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:56,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:56,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:56,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:56,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1428416e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,842 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,842 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:56,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:56,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:56,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:56,849 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:56,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:56,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:56,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:56,849 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:56,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,852 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:56,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:56,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:56,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:56,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:56,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:56,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:56,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:56,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:56,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:56,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:56,962 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:56,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:56,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:56,963 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,963 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:56,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:56,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:56,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:56,971 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/26192a0d501145719a5396ff156d6db0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/26192a0d501145719a5396ff156d6db0 2023-04-24 18:53:56,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362496973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:56,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362496974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362496976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,977 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 26192a0d501145719a5396ff156d6db0(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:56,977 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:56,977 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=10, startTime=1682362436495; duration=0sec 2023-04-24 18:53:56,977 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:56,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362496977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:56,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c02f304[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:56,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:56,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,999 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:57,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:57,002 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,002 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:57,005 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:57,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:57,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:57,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:57,005 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:57,009 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,011 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:57,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:57,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:57,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:57,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,027 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/4d982cbc75ad40cb86a910fa8ef14892 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d982cbc75ad40cb86a910fa8ef14892 2023-04-24 18:53:57,035 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 4d982cbc75ad40cb86a910fa8ef14892(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:57,035 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,035 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=10, startTime=1682362436495; duration=0sec 2023-04-24 18:53:57,035 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:57,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362497064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:57,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:57,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1bae358f942d4581859a9460a0340a0f 2023-04-24 18:53:57,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:57,125 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:57,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:57,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:57,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/e92eb0ca532943fb92c34f1a888f24b9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9 2023-04-24 18:53:57,127 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,127 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,129 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:57,131 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,131 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,131 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:57,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:57,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9, entries=200, sequenceid=749, filesize=14.3 K 2023-04-24 18:53:57,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:57,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:57,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:57,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:57,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/f24b33d899ea46cca6ccc205e2205f79 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79 2023-04-24 18:53:57,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79, entries=150, sequenceid=749, filesize=11.9 K 2023-04-24 18:53:57,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1bae358f942d4581859a9460a0340a0f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f 2023-04-24 18:53:57,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f, entries=150, sequenceid=749, filesize=11.9 K 2023-04-24 18:53:57,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 84739daff3fefab14f3952d64d6f3630 in 501ms, sequenceid=749, compaction requested=true 2023-04-24 18:53:57,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:57,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:57,149 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:57,149 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:57,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:57,151 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:57,151 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:57,151 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,151 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/26192a0d501145719a5396ff156d6db0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=38.9 K 2023-04-24 18:53:57,151 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:57,152 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:57,152 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,152 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 26192a0d501145719a5396ff156d6db0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:57,152 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8b58b4950a474c3db418a2b214b14aec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.6 K 2023-04-24 18:53:57,153 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8b58b4950a474c3db418a2b214b14aec, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:57,153 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting eab576403d8a4214a668869d67f2b424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1682362436380 2023-04-24 18:53:57,153 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 28266ca1576348a7a903421ce4b21271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1682362436380 2023-04-24 18:53:57,153 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e92eb0ca532943fb92c34f1a888f24b9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:57,154 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f24b33d899ea46cca6ccc205e2205f79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:57,163 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:57,169 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:57,179 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,179 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,179 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,179 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:57,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7931bb3f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:57,183 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:57,183 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,184 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/f0bf882882f44ed38a287945b3c5e0dd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/f0bf882882f44ed38a287945b3c5e0dd 2023-04-24 18:53:57,188 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/79988230c3d046e89d3f7f51feaabaf5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/79988230c3d046e89d3f7f51feaabaf5 2023-04-24 18:53:57,192 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into f0bf882882f44ed38a287945b3c5e0dd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:57,192 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,192 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362437149; duration=0sec 2023-04-24 18:53:57,192 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:57,192 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:57,193 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:57,193 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:57,193 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,193 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d982cbc75ad40cb86a910fa8ef14892, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.6 K 2023-04-24 18:53:57,194 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4d982cbc75ad40cb86a910fa8ef14892, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1682362436058 2023-04-24 18:53:57,194 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 79988230c3d046e89d3f7f51feaabaf5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:57,194 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,194 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362437149; duration=0sec 2023-04-24 18:53:57,194 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f805bd8f77664311a979ca2ab229390c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1682362436380 2023-04-24 18:53:57,194 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:57,195 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1bae358f942d4581859a9460a0340a0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:57,201 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:57,218 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/41b4c5f9362e4373be3d3c4832a11719 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/41b4c5f9362e4373be3d3c4832a11719 2023-04-24 18:53:57,224 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 41b4c5f9362e4373be3d3c4832a11719(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:57,224 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,224 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362437149; duration=0sec 2023-04-24 18:53:57,224 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:57,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:57,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:57,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:57,279 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:57,290 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:57,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:57,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:57,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:57,291 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:57,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362497297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362497297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362497298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362497301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/99061943e35d4ff3af81bd6a2b062131 2023-04-24 18:53:57,332 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,334 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362497401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362497401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362497402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362497403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,455 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:57,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:57,455 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,455 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,456 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:57,456 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:57,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:57,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:57,560 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:57,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:57,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:57,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,594 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:57,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:57,596 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:57,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362497603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362497605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362497605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362497606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:57,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ca87c5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:57,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:57,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,655 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,656 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,656 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:57,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:57,698 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:57,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:57,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:57,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:57,698 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:57,732 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,732 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,732 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d6f7f4e869a04d4f905a9e888fe7dd27 2023-04-24 18:53:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/a7c9c386cf7140ac923e18d382a07ace 2023-04-24 18:53:57,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/99061943e35d4ff3af81bd6a2b062131 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131 2023-04-24 18:53:57,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131, entries=150, sequenceid=767, filesize=11.9 K 2023-04-24 18:53:57,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/d6f7f4e869a04d4f905a9e888fe7dd27 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27 2023-04-24 18:53:57,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27, entries=150, sequenceid=767, filesize=11.9 K 2023-04-24 18:53:57,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/a7c9c386cf7140ac923e18d382a07ace as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace 2023-04-24 18:53:57,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace, entries=150, sequenceid=767, filesize=11.9 K 2023-04-24 18:53:57,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 84739daff3fefab14f3952d64d6f3630 in 513ms, sequenceid=767, compaction requested=false 2023-04-24 18:53:57,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,803 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,802 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:57,803 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:57,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:57,903 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:57,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:57,906 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:57,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:57,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:57,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:57,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:53:57,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:57,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:57,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:57,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362497915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362497916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362497917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362497920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/7c4e8ca853384fbcb2acfe5ee52e483d 2023-04-24 18:53:57,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:57,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:57,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:57,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:57,950 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,950 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:57,950 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:57,950 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:57,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bf9143[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:57,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:57,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:57,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:57,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:57,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:58,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362498019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362498019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362498019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362498022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:58,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:58,028 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:58,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:58,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:58,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:58,028 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:58,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:58,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,126 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:58,126 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:58,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362498221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362498221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362498222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362498224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,226 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:58,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:58,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:58,230 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:58,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:58,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:58,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,232 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:58,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:58,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:58,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:58,239 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,239 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:58,239 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:58,239 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:58,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1ed683[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:58,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:58,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:58,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:58,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:58,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:58,252 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:58,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:58,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:58,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:58,252 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:58,254 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,254 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,254 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,254 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,255 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:58,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,265 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:58,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:58,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,265 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/b30d9a9d3f20446f870f2914b4012349 2023-04-24 18:53:58,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:58,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:58,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:58,369 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:58,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:58,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:58,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,372 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:58,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:58,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:58,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:58,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:58,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:58,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:58,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f82749c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:58,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:58,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:58,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:58,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,385 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,385 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:58,388 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:58,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:58,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:58,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:58,388 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:58,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:58,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:58,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:58,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:58,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:58,501 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:58,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:58,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:58,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,503 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:58,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:58,505 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,505 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:58,505 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:58,505 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:58,507 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d042806[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:58,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:58,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:58,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:58,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:58,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:58,512 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:58,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:58,515 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:58,515 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:58,515 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:58,515 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:58,515 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:58,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:58,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:58,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,522 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:58,522 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362498524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362498526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362498527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362498527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,622 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:58,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:58,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:58,626 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:58,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:58,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:58,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:58,640 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,640 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:58,640 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:58,640 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:58,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,643 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bd9d6c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:58,646 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:58,650 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:58,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:58,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:58,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:58,654 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:58,661 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:58,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:58,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:58,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:58,661 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:58,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:58,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:58,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:58,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:58,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:58,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:58,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:58,676 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:58,676 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:58,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/84a6385215914322a6766bdba7dad854 2023-04-24 18:53:58,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/7c4e8ca853384fbcb2acfe5ee52e483d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d 2023-04-24 18:53:58,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d, entries=150, sequenceid=788, filesize=11.9 K 2023-04-24 18:53:58,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/b30d9a9d3f20446f870f2914b4012349 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349 2023-04-24 18:53:58,776 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:58,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:58,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:58,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349, entries=150, sequenceid=788, filesize=11.9 K 2023-04-24 18:53:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:58,783 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:58,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:58,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:58,784 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,784 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/84a6385215914322a6766bdba7dad854 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854 2023-04-24 18:53:58,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:58,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:58,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854, entries=150, sequenceid=788, filesize=11.9 K 2023-04-24 18:53:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:58,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:58,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:58,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:58,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 84739daff3fefab14f3952d64d6f3630 in 891ms, sequenceid=788, compaction requested=true 2023-04-24 18:53:58,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:58,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:58,799 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:58,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:58,799 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:58,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:58,800 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:58,800 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:53:58,800 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,801 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:58,801 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/f0bf882882f44ed38a287945b3c5e0dd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.7 K 2023-04-24 18:53:58,801 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:53:58,801 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,801 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/79988230c3d046e89d3f7f51feaabaf5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.7 K 2023-04-24 18:53:58,801 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f0bf882882f44ed38a287945b3c5e0dd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:58,801 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 79988230c3d046e89d3f7f51feaabaf5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:58,804 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 99061943e35d4ff3af81bd6a2b062131, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1682362436654 2023-04-24 18:53:58,804 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d6f7f4e869a04d4f905a9e888fe7dd27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1682362436654 2023-04-24 18:53:58,805 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7c4e8ca853384fbcb2acfe5ee52e483d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:53:58,805 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b30d9a9d3f20446f870f2914b4012349, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:53:58,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:58,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:58,818 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:58,826 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:58,828 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,828 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:58,828 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:58,829 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:53:58,874 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/cc84245a2eda41e6aa69bdec7233f0fa as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/cc84245a2eda41e6aa69bdec7233f0fa 2023-04-24 18:53:58,888 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into cc84245a2eda41e6aa69bdec7233f0fa(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:58,888 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:58,888 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362438799; duration=0sec 2023-04-24 18:53:58,888 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/3173d6cb4dcd429199ccd33ee00bc82a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/3173d6cb4dcd429199ccd33ee00bc82a 2023-04-24 18:53:58,888 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:53:58,888 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:53:58,890 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:53:58,890 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:53:58,890 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:58,891 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/41b4c5f9362e4373be3d3c4832a11719, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.7 K 2023-04-24 18:53:58,891 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 41b4c5f9362e4373be3d3c4832a11719, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1682362436536 2023-04-24 18:53:58,891 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a7c9c386cf7140ac923e18d382a07ace, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1682362436654 2023-04-24 18:53:58,892 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 84a6385215914322a6766bdba7dad854, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:53:58,899 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 3173d6cb4dcd429199ccd33ee00bc82a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:58,899 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:58,899 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362438799; duration=0sec 2023-04-24 18:53:58,899 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:58,914 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:53:59,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:59,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362439043, max timestamp=1682362439053Num uniques 859; 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362439043, max timestamp=1682362439053Num uniques 859; 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362439043, max timestamp=1682362439053Num uniques 859; 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:53:59,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362499069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362499070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362499072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362499072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35444 deadline: 1682362499086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,089 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:53:59,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362499173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362499173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362499177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362499177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,298 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/e27d54c90f284d8988397761c879fd10 2023-04-24 18:53:59,310 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/51e8e012be1147f1b903a738aa4cb79f 2023-04-24 18:53:59,322 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/d22e130b4bd34cbca7734ed9fcf0099f 2023-04-24 18:53:59,327 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/e27d54c90f284d8988397761c879fd10 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10 2023-04-24 18:53:59,332 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10, entries=150, sequenceid=799, filesize=11.9 K 2023-04-24 18:53:59,333 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/51e8e012be1147f1b903a738aa4cb79f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f 2023-04-24 18:53:59,341 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f, entries=150, sequenceid=799, filesize=11.9 K 2023-04-24 18:53:59,342 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/d22e130b4bd34cbca7734ed9fcf0099f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f 2023-04-24 18:53:59,350 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f, entries=150, sequenceid=799, filesize=11.9 K 2023-04-24 18:53:59,351 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 84739daff3fefab14f3952d64d6f3630 in 523ms, sequenceid=799, compaction requested=false 2023-04-24 18:53:59,351 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:59,351 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:59,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:59,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:59,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7caae116[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:59,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,363 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:59,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:59,366 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,370 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/d35c76282eb04d4f8af7b62f29f85375 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d35c76282eb04d4f8af7b62f29f85375 2023-04-24 18:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:59,379 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:59,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:59,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:59,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:59,379 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:59,380 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into d35c76282eb04d4f8af7b62f29f85375(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:53:59,380 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:53:59,380 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362438799; duration=0sec 2023-04-24 18:53:59,380 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:53:59,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:53:59,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:53:59,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:53:59,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,400 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362499397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362499399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362499399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362499401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:59,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:59,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:59,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:59,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:59,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/d16eaf78cc5846389294c4cfbf72aa6b 2023-04-24 18:53:59,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362499502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:59,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:59,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362499504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362499504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362499505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:59,509 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:59,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:59,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:59,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:59,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:59,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:59,513 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:59,513 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:59,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a05c8b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:59,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:59,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:59,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:59,520 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:59,522 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:59,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:59,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:59,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:59,522 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:59,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:59,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:59,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:59,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:59,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:59,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:59,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:59,640 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:59,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:59,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:59,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,643 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,643 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:59,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:59,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:59,645 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,645 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:59,645 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:59,645 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:59,648 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c880a81[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:59,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:59,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:59,656 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:59,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:59,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:59,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:59,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:59,661 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:59,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:59,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:59,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:59,661 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:59,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,664 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:59,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,669 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,669 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:59,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:59,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:59,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362499706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362499709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362499709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:53:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362499711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:59,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:59,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:59,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:59,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:59,772 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:59,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:59,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:59,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,774 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,774 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,774 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,774 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:59,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:59,776 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,776 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:59,776 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:59,776 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:59,778 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f3a681[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:59,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:59,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:59,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:59,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:59,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:59,787 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:59,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:59,790 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:59,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:59,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:59,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:59,790 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:59,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,793 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,794 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:53:59,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:59,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:59,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,797 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:59,797 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:59,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/88ad6f571c2946378890ca3312db85b3 2023-04-24 18:53:59,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:53:59,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:53:59,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:53:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:53:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:53:59,901 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:53:59,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:53:59,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:53:59,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:53:59,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:53:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7a57f8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:53:59,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:53:59,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,911 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:53:59,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:53:59,913 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:53:59,916 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:53:59,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:53:59,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:53:59,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:53:59,916 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:53:59,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:53:59,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,919 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,921 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:53:59,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:53:59,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:53:59,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:53:59,923 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:53:59,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:53:59,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:53:59,924 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:53:59,924 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,924 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:53:59,924 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:53:59,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:53:59,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:53:59,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:00,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35456 deadline: 1682362500010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35514 deadline: 1682362500011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35498 deadline: 1682362500012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35508 deadline: 1682362500012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:00,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:00,027 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:00,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:00,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:00,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:00,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:00,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:00,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:00,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:00,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:00,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:00,031 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,031 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:54:00,031 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:54:00,031 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:00,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:00,033 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686c1038[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:00,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:00,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:00,037 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,037 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:00,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:00,040 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:00,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:00,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:00,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:00,041 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:00,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:00,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,044 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,045 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:00,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:00,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:00,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:00,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:00,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:00,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:00,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:00,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:00,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:00,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:00,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:00,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:00,153 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:00,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:00,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:00,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:00,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:00,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:00,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:00,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:00,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:54:00,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:54:00,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:00,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:00,162 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,162 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:00,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ccc3b6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:00,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:00,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:00,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:00,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:00,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:00,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:00,174 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:00,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:00,174 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:00,175 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,175 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:00,175 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:00,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:00,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:00,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:00,180 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:00,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:00,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:00,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:00,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:00,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/671fcd674648454986a48b3e9ba89b28 2023-04-24 18:54:00,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/d16eaf78cc5846389294c4cfbf72aa6b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b 2023-04-24 18:54:00,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b, entries=200, sequenceid=839, filesize=14.3 K 2023-04-24 18:54:00,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/88ad6f571c2946378890ca3312db85b3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3 2023-04-24 18:54:00,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3, entries=150, sequenceid=839, filesize=11.9 K 2023-04-24 18:54:00,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/671fcd674648454986a48b3e9ba89b28 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28 2023-04-24 18:54:00,281 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:00,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:00,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28, entries=150, sequenceid=839, filesize=11.9 K 2023-04-24 18:54:00,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 84739daff3fefab14f3952d64d6f3630 in 906ms, sequenceid=839, compaction requested=true 2023-04-24 18:54:00,287 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:00,287 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:00,287 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:00,288 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:00,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:00,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:00,290 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:00,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:00,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:00,291 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:00,291 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:00,291 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/B is initiating minor compaction (all files) 2023-04-24 18:54:00,291 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/A is initiating minor compaction (all files) 2023-04-24 18:54:00,291 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/B in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,291 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/A in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,291 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/cc84245a2eda41e6aa69bdec7233f0fa, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.8 K 2023-04-24 18:54:00,291 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/3173d6cb4dcd429199ccd33ee00bc82a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=39.1 K 2023-04-24 18:54:00,291 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting cc84245a2eda41e6aa69bdec7233f0fa, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:54:00,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,292 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 51e8e012be1147f1b903a738aa4cb79f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1682362437910 2023-04-24 18:54:00,292 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3173d6cb4dcd429199ccd33ee00bc82a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:54:00,293 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 88ad6f571c2946378890ca3312db85b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682362439069 2023-04-24 18:54:00,293 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e27d54c90f284d8988397761c879fd10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1682362437910 2023-04-24 18:54:00,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:00,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:00,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:00,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,294 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d16eaf78cc5846389294c4cfbf72aa6b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682362439068 2023-04-24 18:54:00,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:00,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:00,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:00,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. started... 2023-04-24 18:54:00,313 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:54:00,313 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:54:00,314 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:00,314 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:54:00,315 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:00,315 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:54:00,315 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:00,325 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#B#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:00,328 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#A#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:00,367 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/ac32bb13753b421ca683de327906a23d 2023-04-24 18:54:00,375 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/0a8a648f30284e16904295a65be57de3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0a8a648f30284e16904295a65be57de3 2023-04-24 18:54:00,391 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/A of 84739daff3fefab14f3952d64d6f3630 into 0a8a648f30284e16904295a65be57de3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:00,391 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:00,392 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/A, priority=13, startTime=1682362440287; duration=0sec 2023-04-24 18:54:00,392 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:00,392 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:00,394 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:00,395 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 84739daff3fefab14f3952d64d6f3630/C is initiating minor compaction (all files) 2023-04-24 18:54:00,395 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 84739daff3fefab14f3952d64d6f3630/C in TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:00,395 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d35c76282eb04d4f8af7b62f29f85375, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp, totalSize=36.8 K 2023-04-24 18:54:00,395 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d35c76282eb04d4f8af7b62f29f85375, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1682362437297 2023-04-24 18:54:00,395 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d22e130b4bd34cbca7734ed9fcf0099f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1682362437910 2023-04-24 18:54:00,396 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 671fcd674648454986a48b3e9ba89b28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682362439069 2023-04-24 18:54:00,417 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 84739daff3fefab14f3952d64d6f3630#C#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:00,493 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:59583 2023-04-24 18:54:00,493 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,495 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:59583 2023-04-24 18:54:00,495 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,496 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:59583 2023-04-24 18:54:00,496 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,497 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1423ce11 to 127.0.0.1:59583 2023-04-24 18:54:00,497 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:54:00,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. as already flushing 2023-04-24 18:54:00,518 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:59583 2023-04-24 18:54:00,518 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,521 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60b8c4c7 to 127.0.0.1:59583 2023-04-24 18:54:00,521 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,523 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:59583 2023-04-24 18:54:00,523 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,529 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:59583 2023-04-24 18:54:00,529 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:00,760 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/933c1a299a2c4cbba84f71c532a3f8cb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/933c1a299a2c4cbba84f71c532a3f8cb 2023-04-24 18:54:00,765 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/B of 84739daff3fefab14f3952d64d6f3630 into 933c1a299a2c4cbba84f71c532a3f8cb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:00,765 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:00,765 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/B, priority=13, startTime=1682362440287; duration=0sec 2023-04-24 18:54:00,765 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:00,793 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/b3ae49b0392e4211b3821430add61b53 2023-04-24 18:54:00,872 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/2d2e30b950d448c4b6210c87262911bc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/2d2e30b950d448c4b6210c87262911bc 2023-04-24 18:54:00,878 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 84739daff3fefab14f3952d64d6f3630/C of 84739daff3fefab14f3952d64d6f3630 into 2d2e30b950d448c4b6210c87262911bc(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:00,878 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:00,878 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630., storeName=84739daff3fefab14f3952d64d6f3630/C, priority=13, startTime=1682362440288; duration=0sec 2023-04-24 18:54:00,879 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:01,205 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=848 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/6d6fe4e4df5c43cdb519906754df2c70 2023-04-24 18:54:01,211 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/ac32bb13753b421ca683de327906a23d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ac32bb13753b421ca683de327906a23d 2023-04-24 18:54:01,215 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ac32bb13753b421ca683de327906a23d, entries=150, sequenceid=848, filesize=11.9 K 2023-04-24 18:54:01,216 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/b3ae49b0392e4211b3821430add61b53 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b3ae49b0392e4211b3821430add61b53 2023-04-24 18:54:01,220 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b3ae49b0392e4211b3821430add61b53, entries=150, sequenceid=848, filesize=11.9 K 2023-04-24 18:54:01,221 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/6d6fe4e4df5c43cdb519906754df2c70 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/6d6fe4e4df5c43cdb519906754df2c70 2023-04-24 18:54:01,225 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/6d6fe4e4df5c43cdb519906754df2c70, entries=150, sequenceid=848, filesize=11.9 K 2023-04-24 18:54:01,226 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=26.84 KB/27480 for 84739daff3fefab14f3952d64d6f3630 in 913ms, sequenceid=848, compaction requested=false 2023-04-24 18:54:01,226 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:01,226 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:01,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:01,229 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:01,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:01,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1887a800[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:01,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:01,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,232 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:01,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:01,234 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:01,236 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:01,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:01,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:01,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:01,236 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:01,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:01,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:01,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:01,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:01,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:01,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:01,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:01,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:01,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:01,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:03,111 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:59583 2023-04-24 18:54:03,111 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:03,111 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 189 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 208 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4422 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4204 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1898 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5691 rows 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1881 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5642 rows 2023-04-24 18:54:03,112 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:54:03,112 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55e5fd59 to 127.0.0.1:59583 2023-04-24 18:54:03,112 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:03,116 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:54:03,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:54:03,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:03,131 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362443131"}]},"ts":"1682362443131"} 2023-04-24 18:54:03,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:03,132 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:54:03,133 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:54:03,135 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, UNASSIGN}] 2023-04-24 18:54:03,136 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, UNASSIGN 2023-04-24 18:54:03,137 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=84739daff3fefab14f3952d64d6f3630, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:03,137 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362443137"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362443137"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362443137"}]},"ts":"1682362443137"} 2023-04-24 18:54:03,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:03,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:03,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:54:03,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 84739daff3fefab14f3952d64d6f3630, disabling compactions & flushes 2023-04-24 18:54:03,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:03,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:03,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. after waiting 0 ms 2023-04-24 18:54:03,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:03,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 84739daff3fefab14f3952d64d6f3630 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=A 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=B 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 84739daff3fefab14f3952d64d6f3630, store=C 2023-04-24 18:54:03,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:03,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/60af131d58114029a2695fe30e1b5b21 2023-04-24 18:54:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:04,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/ce43671e37394214994e8ca5ac75ccad 2023-04-24 18:54:04,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:04,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1885972403bb474ababed3890bb29ff6 2023-04-24 18:54:04,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/A/60af131d58114029a2695fe30e1b5b21 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/60af131d58114029a2695fe30e1b5b21 2023-04-24 18:54:04,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/60af131d58114029a2695fe30e1b5b21, entries=150, sequenceid=859, filesize=11.9 K 2023-04-24 18:54:04,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/B/ce43671e37394214994e8ca5ac75ccad as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/ce43671e37394214994e8ca5ac75ccad 2023-04-24 18:54:04,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/ce43671e37394214994e8ca5ac75ccad, entries=150, sequenceid=859, filesize=11.9 K 2023-04-24 18:54:04,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/.tmp/C/1885972403bb474ababed3890bb29ff6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1885972403bb474ababed3890bb29ff6 2023-04-24 18:54:04,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1885972403bb474ababed3890bb29ff6, entries=150, sequenceid=859, filesize=11.9 K 2023-04-24 18:54:04,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 84739daff3fefab14f3952d64d6f3630 in 1254ms, sequenceid=859, compaction requested=true 2023-04-24 18:54:04,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ce55959d03fe4287b2e9caf366f06c68, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2f5fa87b883a4c69bb30d27b9230493a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0d953ab5e4404ad38cc8c78c0034f6ff, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7e62f5982c4143d8bcead9ae718d04de, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c656866a6954b5f995a029bea8a9a5a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/750ef5b8168f43a4b4e5a2a091fdd013, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/26192a0d501145719a5396ff156d6db0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/f0bf882882f44ed38a287945b3c5e0dd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/3173d6cb4dcd429199ccd33ee00bc82a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b] to archive 2023-04-24 18:54:04,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:04,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7877532514be4bfe860cba1edb07cff2 2023-04-24 18:54:04,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9b28a2e214244347a861ddccd3d3b254 2023-04-24 18:54:04,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/84f8f0ceee07439988c1ca6c4d797e4e 2023-04-24 18:54:04,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ce55959d03fe4287b2e9caf366f06c68 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ce55959d03fe4287b2e9caf366f06c68 2023-04-24 18:54:04,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/fba8f3f1a87b4064808d695e958d7889 2023-04-24 18:54:04,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/c5e5667307464b3aab3e1d7a3d549e8e 2023-04-24 18:54:04,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2f5fa87b883a4c69bb30d27b9230493a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2f5fa87b883a4c69bb30d27b9230493a 2023-04-24 18:54:04,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/03db3d6ecd2941c780c188d336a6f983 2023-04-24 18:54:04,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/672f65468f064027a949391a64537a14 2023-04-24 18:54:04,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0d953ab5e4404ad38cc8c78c0034f6ff to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0d953ab5e4404ad38cc8c78c0034f6ff 2023-04-24 18:54:04,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/769dc9377be6411e96657fe3fc5e43c8 2023-04-24 18:54:04,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2cd909b6dae94dd9bb6caead0d4132d5 2023-04-24 18:54:04,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0e5dc240368543fcbce11d34190e0248 2023-04-24 18:54:04,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7e62f5982c4143d8bcead9ae718d04de to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7e62f5982c4143d8bcead9ae718d04de 2023-04-24 18:54:04,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c1d607e33c34e39945154b3665e10c2 2023-04-24 18:54:04,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/98bcedad954e4240b579a6106af3229a 2023-04-24 18:54:04,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/08eeb2fdf56e4df1aa4d66e0e394eea2 2023-04-24 18:54:04,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c656866a6954b5f995a029bea8a9a5a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/6c656866a6954b5f995a029bea8a9a5a 2023-04-24 18:54:04,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/1a4076d6ecfe4f47ad93221d0475d073 2023-04-24 18:54:04,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/750ef5b8168f43a4b4e5a2a091fdd013 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/750ef5b8168f43a4b4e5a2a091fdd013 2023-04-24 18:54:04,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/8a94782333f84c47bb67e36e17b9004c 2023-04-24 18:54:04,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/95e447c3d89d4906984183dcfe36e772 2023-04-24 18:54:04,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/9d7d5f7bafe1465f9e04c1e9f459c86a 2023-04-24 18:54:04,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d8d695d56d2f4f02b5dabe35e17ad6d7 2023-04-24 18:54:04,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/893e246f2c904bebb80daea11f0a2fa4 2023-04-24 18:54:04,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/26192a0d501145719a5396ff156d6db0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/26192a0d501145719a5396ff156d6db0 2023-04-24 18:54:04,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/2a83d45527784f4faf9928f40e0a77ae 2023-04-24 18:54:04,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/eab576403d8a4214a668869d67f2b424 2023-04-24 18:54:04,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e92eb0ca532943fb92c34f1a888f24b9 2023-04-24 18:54:04,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/f0bf882882f44ed38a287945b3c5e0dd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/f0bf882882f44ed38a287945b3c5e0dd 2023-04-24 18:54:04,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/99061943e35d4ff3af81bd6a2b062131 2023-04-24 18:54:04,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/3173d6cb4dcd429199ccd33ee00bc82a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/3173d6cb4dcd429199ccd33ee00bc82a 2023-04-24 18:54:04,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/7c4e8ca853384fbcb2acfe5ee52e483d 2023-04-24 18:54:04,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/e27d54c90f284d8988397761c879fd10 2023-04-24 18:54:04,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/d16eaf78cc5846389294c4cfbf72aa6b 2023-04-24 18:54:04,656 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/82c1c1de02c84f8aaea9da8814e336f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8efb055dff3b47a0b6515492fcfdc8d0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/84367fcc7bd64500ad7103f07d38335e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2247842d7dbc4f8481ebf88e4c1bfb9a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/c0c06acf136b4910a331d75eee2dc24d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/0aa55e500d6e4afca856241e090782e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8b58b4950a474c3db418a2b214b14aec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/79988230c3d046e89d3f7f51feaabaf5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/cc84245a2eda41e6aa69bdec7233f0fa, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3] to archive 2023-04-24 18:54:04,657 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:04,659 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7db4cfd9b07f426dbd6400ca79af518b 2023-04-24 18:54:04,660 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e80f3f1c41dc4ca7b321371814a228eb 2023-04-24 18:54:04,661 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/82c1c1de02c84f8aaea9da8814e336f3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/82c1c1de02c84f8aaea9da8814e336f3 2023-04-24 18:54:04,662 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f6d6e84508ae4d69acd82684a9ba9e3c 2023-04-24 18:54:04,663 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/18db87669eb340b19beca1a5107ee3a4 2023-04-24 18:54:04,664 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8efb055dff3b47a0b6515492fcfdc8d0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8efb055dff3b47a0b6515492fcfdc8d0 2023-04-24 18:54:04,665 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/3970df994e3043df880acfe1e3b8d8e3 2023-04-24 18:54:04,666 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5a1038fcfd2d4834b42ff4ed976e3273 2023-04-24 18:54:04,667 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/84367fcc7bd64500ad7103f07d38335e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/84367fcc7bd64500ad7103f07d38335e 2023-04-24 18:54:04,668 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d0aafe430e824669baf25c4092bda7cc 2023-04-24 18:54:04,669 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2eb17ddd41114345961dc51c45145036 2023-04-24 18:54:04,670 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/feabec8648894320b99196fa79643951 2023-04-24 18:54:04,671 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2247842d7dbc4f8481ebf88e4c1bfb9a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/2247842d7dbc4f8481ebf88e4c1bfb9a 2023-04-24 18:54:04,672 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/e9ce963f1ed844908ad5d7808f82e365 2023-04-24 18:54:04,673 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/5484a803c2234d8189eb7bde744ea41d 2023-04-24 18:54:04,674 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/6ea3fc3b6eb14ec493af7ee7def13993 2023-04-24 18:54:04,675 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/c0c06acf136b4910a331d75eee2dc24d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/c0c06acf136b4910a331d75eee2dc24d 2023-04-24 18:54:04,676 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8c2d1baa6ebd49008a375693f056779f 2023-04-24 18:54:04,677 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/a494e3d8305545a69c004fd11af96222 2023-04-24 18:54:04,678 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/0aa55e500d6e4afca856241e090782e9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/0aa55e500d6e4afca856241e090782e9 2023-04-24 18:54:04,679 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d3bc2e96a9194771b2e97e5fa6db64fc 2023-04-24 18:54:04,681 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/4457cb43e8f94a8295c693644d2ad5e7 2023-04-24 18:54:04,682 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/785bcd351c094bc7bea099f74e7fbefa 2023-04-24 18:54:04,683 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/afed1d9a613d4658870c0863fa0593ca 2023-04-24 18:54:04,684 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f42d0d41adf04f8298df1bce6301bc74 2023-04-24 18:54:04,685 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8b58b4950a474c3db418a2b214b14aec to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/8b58b4950a474c3db418a2b214b14aec 2023-04-24 18:54:04,686 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/7072a3f2312f467aa154712895dd4383 2023-04-24 18:54:04,687 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/28266ca1576348a7a903421ce4b21271 2023-04-24 18:54:04,688 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/79988230c3d046e89d3f7f51feaabaf5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/79988230c3d046e89d3f7f51feaabaf5 2023-04-24 18:54:04,690 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/f24b33d899ea46cca6ccc205e2205f79 2023-04-24 18:54:04,691 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/d6f7f4e869a04d4f905a9e888fe7dd27 2023-04-24 18:54:04,692 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/cc84245a2eda41e6aa69bdec7233f0fa to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/cc84245a2eda41e6aa69bdec7233f0fa 2023-04-24 18:54:04,693 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b30d9a9d3f20446f870f2914b4012349 2023-04-24 18:54:04,694 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/51e8e012be1147f1b903a738aa4cb79f 2023-04-24 18:54:04,696 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/88ad6f571c2946378890ca3312db85b3 2023-04-24 18:54:04,699 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/866a4e271b6b4b8ab6629c0544743333, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c4d66b80845c45b794fb691c3582b06f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/5dd4f20b5f9f4d9dbce75071acb26882, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f09eb8c47fec493299d96c3f6b4b29f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/375e6d01bfcd4092ab50e1772ceb747d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4e099128c5904e99b2d677c73233e118, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d982cbc75ad40cb86a910fa8ef14892, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/41b4c5f9362e4373be3d3c4832a11719, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d35c76282eb04d4f8af7b62f29f85375, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28] to archive 2023-04-24 18:54:04,700 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:04,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/9c89e84ec7e5411d8fe6f778ca7345cf 2023-04-24 18:54:04,702 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3c93efaeab4644889d832bb4a5b01193 2023-04-24 18:54:04,703 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/866a4e271b6b4b8ab6629c0544743333 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/866a4e271b6b4b8ab6629c0544743333 2023-04-24 18:54:04,704 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7920f0b06920477983e1bc775039ab7c 2023-04-24 18:54:04,705 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d43ab62d4554c7389ebcf51ceeb177a 2023-04-24 18:54:04,706 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c4d66b80845c45b794fb691c3582b06f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c4d66b80845c45b794fb691c3582b06f 2023-04-24 18:54:04,708 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7a7bdb00ea8b4966b8e0b91ac2fa4f17 2023-04-24 18:54:04,709 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1456e675e0e941e8884be72e052194b0 2023-04-24 18:54:04,710 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/5dd4f20b5f9f4d9dbce75071acb26882 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/5dd4f20b5f9f4d9dbce75071acb26882 2023-04-24 18:54:04,711 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/92028617ecf241c1b91b489faeee7ebd 2023-04-24 18:54:04,712 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/bf6ff3e554b5471c990df757bfe25733 2023-04-24 18:54:04,713 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/62b6111110374ef1b6fe6e5d678090fd 2023-04-24 18:54:04,714 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f09eb8c47fec493299d96c3f6b4b29f3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f09eb8c47fec493299d96c3f6b4b29f3 2023-04-24 18:54:04,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0ec491c244a942459ba1d838d4d9fe0f 2023-04-24 18:54:04,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/33d09dc51b0943ac82c23d64101d8f07 2023-04-24 18:54:04,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7d021c2232a54e839b2c013e90fde8bd 2023-04-24 18:54:04,718 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/375e6d01bfcd4092ab50e1772ceb747d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/375e6d01bfcd4092ab50e1772ceb747d 2023-04-24 18:54:04,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/c8d08ed0b27045e481908feb188e659a 2023-04-24 18:54:04,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3f2ca82cc2a841a287c77e83bcb197c4 2023-04-24 18:54:04,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4e099128c5904e99b2d677c73233e118 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4e099128c5904e99b2d677c73233e118 2023-04-24 18:54:04,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/7e75b6454022431bbf4128466bbc9467 2023-04-24 18:54:04,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/3b12bb5ab39f44fd9b27fb4db519a8b4 2023-04-24 18:54:04,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/0abb0187481948e8bacecf7b48f746ac 2023-04-24 18:54:04,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/07a452cf622040aa897c57b9fe93c10c 2023-04-24 18:54:04,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/55ff2264cd2c4413b78c250fc14d4625 2023-04-24 18:54:04,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d982cbc75ad40cb86a910fa8ef14892 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/4d982cbc75ad40cb86a910fa8ef14892 2023-04-24 18:54:04,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1f963be369634aa2bd8a2e6196ff7100 2023-04-24 18:54:04,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/f805bd8f77664311a979ca2ab229390c 2023-04-24 18:54:04,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/41b4c5f9362e4373be3d3c4832a11719 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/41b4c5f9362e4373be3d3c4832a11719 2023-04-24 18:54:04,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1bae358f942d4581859a9460a0340a0f 2023-04-24 18:54:04,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/a7c9c386cf7140ac923e18d382a07ace 2023-04-24 18:54:04,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d35c76282eb04d4f8af7b62f29f85375 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d35c76282eb04d4f8af7b62f29f85375 2023-04-24 18:54:04,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/84a6385215914322a6766bdba7dad854 2023-04-24 18:54:04,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/d22e130b4bd34cbca7734ed9fcf0099f 2023-04-24 18:54:04,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/671fcd674648454986a48b3e9ba89b28 2023-04-24 18:54:04,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/recovered.edits/862.seqid, newMaxSeqId=862, maxSeqId=1 2023-04-24 18:54:04,747 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630. 2023-04-24 18:54:04,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 84739daff3fefab14f3952d64d6f3630: 2023-04-24 18:54:04,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 84739daff3fefab14f3952d64d6f3630 2023-04-24 18:54:04,750 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=84739daff3fefab14f3952d64d6f3630, regionState=CLOSED 2023-04-24 18:54:04,750 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362444750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362444750"}]},"ts":"1682362444750"} 2023-04-24 18:54:04,753 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-24 18:54:04,753 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 84739daff3fefab14f3952d64d6f3630, server=jenkins-hbase4.apache.org,37593,1682362415680 in 1.6130 sec 2023-04-24 18:54:04,755 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-24 18:54:04,755 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=84739daff3fefab14f3952d64d6f3630, UNASSIGN in 1.6180 sec 2023-04-24 18:54:04,756 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362444756"}]},"ts":"1682362444756"} 2023-04-24 18:54:04,757 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:54:04,759 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:54:04,761 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6370 sec 2023-04-24 18:54:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 18:54:05,236 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-24 18:54:05,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:54:05,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,250 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,251 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 18:54:05,256 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:54:05,259 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/recovered.edits] 2023-04-24 18:54:05,262 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0a8a648f30284e16904295a65be57de3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/0a8a648f30284e16904295a65be57de3 2023-04-24 18:54:05,263 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/60af131d58114029a2695fe30e1b5b21 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/60af131d58114029a2695fe30e1b5b21 2023-04-24 18:54:05,265 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ac32bb13753b421ca683de327906a23d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/A/ac32bb13753b421ca683de327906a23d 2023-04-24 18:54:05,267 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/933c1a299a2c4cbba84f71c532a3f8cb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/933c1a299a2c4cbba84f71c532a3f8cb 2023-04-24 18:54:05,269 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b3ae49b0392e4211b3821430add61b53 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/b3ae49b0392e4211b3821430add61b53 2023-04-24 18:54:05,270 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/ce43671e37394214994e8ca5ac75ccad to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/B/ce43671e37394214994e8ca5ac75ccad 2023-04-24 18:54:05,273 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1885972403bb474ababed3890bb29ff6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/1885972403bb474ababed3890bb29ff6 2023-04-24 18:54:05,275 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/2d2e30b950d448c4b6210c87262911bc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/2d2e30b950d448c4b6210c87262911bc 2023-04-24 18:54:05,276 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/6d6fe4e4df5c43cdb519906754df2c70 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/C/6d6fe4e4df5c43cdb519906754df2c70 2023-04-24 18:54:05,279 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/recovered.edits/862.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630/recovered.edits/862.seqid 2023-04-24 18:54:05,280 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/84739daff3fefab14f3952d64d6f3630 2023-04-24 18:54:05,280 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:54:05,282 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,303 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:54:05,345 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:54:05,346 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,346 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:54:05,347 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362445347"}]},"ts":"9223372036854775807"} 2023-04-24 18:54:05,349 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:54:05,349 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 84739daff3fefab14f3952d64d6f3630, NAME => 'TestAcidGuarantees,,1682362419223.84739daff3fefab14f3952d64d6f3630.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:54:05,350 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:54:05,350 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362445350"}]},"ts":"9223372036854775807"} 2023-04-24 18:54:05,352 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:54:05,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 18:54:05,354 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,355 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 111 msec 2023-04-24 18:54:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 18:54:05,554 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-24 18:54:05,571 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 208) Potentially hanging thread: (jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-966207702_15 at /127.0.0.1:45300 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28481538-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28481538-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28481538-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28481538-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:37593-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=589 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=442 (was 352) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4488 (was 5176) 2023-04-24 18:54:05,581 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=589, MaxFileDescriptor=60000, SystemLoadAverage=442, ProcessCount=170, AvailableMemoryMB=4487 2023-04-24 18:54:05,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:54:05,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:54:05,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:05,586 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:54:05,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-24 18:54:05,587 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:54:05,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:05,589 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:05,590 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f empty. 2023-04-24 18:54:05,590 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:05,590 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:54:05,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:05,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:06,000 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:54:06,002 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 103750326d6eb6a8536f7e1c9d4b7a6f, NAME => 'TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:54:06,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:06,409 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:54:06,409 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 103750326d6eb6a8536f7e1c9d4b7a6f, disabling compactions & flushes 2023-04-24 18:54:06,409 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,409 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,409 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. after waiting 0 ms 2023-04-24 18:54:06,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,410 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:06,412 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:54:06,413 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362446413"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362446413"}]},"ts":"1682362446413"} 2023-04-24 18:54:06,414 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:54:06,415 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:54:06,415 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362446415"}]},"ts":"1682362446415"} 2023-04-24 18:54:06,416 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:54:06,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, ASSIGN}] 2023-04-24 18:54:06,422 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, ASSIGN 2023-04-24 18:54:06,423 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:54:06,574 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:06,574 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362446574"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362446574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362446574"}]},"ts":"1682362446574"} 2023-04-24 18:54:06,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:06,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:06,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 103750326d6eb6a8536f7e1c9d4b7a6f, NAME => 'TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:54:06,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:54:06,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,733 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,735 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A 2023-04-24 18:54:06,735 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A 2023-04-24 18:54:06,735 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:06,735 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName A 2023-04-24 18:54:06,736 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:06,736 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,737 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B 2023-04-24 18:54:06,737 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B 2023-04-24 18:54:06,737 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:06,738 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName B 2023-04-24 18:54:06,738 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:06,738 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,739 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C 2023-04-24 18:54:06,739 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C 2023-04-24 18:54:06,739 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:06,740 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName C 2023-04-24 18:54:06,740 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:06,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:54:06,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:54:06,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 103750326d6eb6a8536f7e1c9d4b7a6f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67828466, jitterRate=0.010722905397415161}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:54:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:06,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., pid=18, masterSystemTime=1682362446728 2023-04-24 18:54:06,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:06,750 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:06,750 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362446750"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362446750"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362446750"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362446750"}]},"ts":"1682362446750"} 2023-04-24 18:54:06,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-24 18:54:06,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 175 msec 2023-04-24 18:54:06,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-24 18:54:06,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, ASSIGN in 333 msec 2023-04-24 18:54:06,755 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:54:06,756 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362446755"}]},"ts":"1682362446755"} 2023-04-24 18:54:06,757 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:54:06,759 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:54:06,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1750 sec 2023-04-24 18:54:07,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 18:54:07,693 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-24 18:54:07,694 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x324dc814 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:07,705 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a301dcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:07,707 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:07,717 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:07,719 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:54:07,723 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:54:07,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:54:07,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:54:07,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:08,172 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:54:08,173 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:54:08,173 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:54:08,177 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 18:54:08,189 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, REOPEN/MOVE}] 2023-04-24 18:54:08,190 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, REOPEN/MOVE 2023-04-24 18:54:08,190 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,190 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362448190"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362448190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362448190"}]},"ts":"1682362448190"} 2023-04-24 18:54:08,192 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:08,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 103750326d6eb6a8536f7e1c9d4b7a6f, disabling compactions & flushes 2023-04-24 18:54:08,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. after waiting 0 ms 2023-04-24 18:54:08,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 18:54:08,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:08,349 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 103750326d6eb6a8536f7e1c9d4b7a6f to self. 2023-04-24 18:54:08,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,351 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=CLOSED 2023-04-24 18:54:08,351 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362448351"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362448351"}]},"ts":"1682362448351"} 2023-04-24 18:54:08,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-24 18:54:08,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 160 msec 2023-04-24 18:54:08,354 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=true 2023-04-24 18:54:08,505 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,505 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362448505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362448505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362448505"}]},"ts":"1682362448505"} 2023-04-24 18:54:08,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:08,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 103750326d6eb6a8536f7e1c9d4b7a6f, NAME => 'TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:54:08,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:54:08,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,665 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,666 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A 2023-04-24 18:54:08,666 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A 2023-04-24 18:54:08,666 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:08,669 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName A 2023-04-24 18:54:08,671 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:08,671 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,672 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B 2023-04-24 18:54:08,672 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B 2023-04-24 18:54:08,672 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:08,673 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName B 2023-04-24 18:54:08,673 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:08,673 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,674 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C 2023-04-24 18:54:08,674 DEBUG [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C 2023-04-24 18:54:08,674 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:08,675 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 103750326d6eb6a8536f7e1c9d4b7a6f columnFamilyName C 2023-04-24 18:54:08,675 INFO [StoreOpener-103750326d6eb6a8536f7e1c9d4b7a6f-1] regionserver.HStore(310): Store=103750326d6eb6a8536f7e1c9d4b7a6f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:08,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:54:08,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 103750326d6eb6a8536f7e1c9d4b7a6f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65412289, jitterRate=-0.025280937552452087}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:54:08,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:08,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., pid=23, masterSystemTime=1682362448658 2023-04-24 18:54:08,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,683 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,684 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362448683"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362448683"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362448683"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362448683"}]},"ts":"1682362448683"} 2023-04-24 18:54:08,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-24 18:54:08,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 179 msec 2023-04-24 18:54:08,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-24 18:54:08,688 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, REOPEN/MOVE in 497 msec 2023-04-24 18:54:08,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-24 18:54:08,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-04-24 18:54:08,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 952 msec 2023-04-24 18:54:08,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-24 18:54:08,700 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,705 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,706 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,710 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,710 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,716 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,716 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,722 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,722 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,725 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,726 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,730 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,730 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,734 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,734 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,738 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,739 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x42d325f4 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:08,742 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc1d73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:08,749 DEBUG [hconnection-0x7eaa2fec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,749 DEBUG [hconnection-0x3b379df9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,751 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:08,758 DEBUG [hconnection-0x6e86b11c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,758 DEBUG [hconnection-0x5e9cf749-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,758 DEBUG [hconnection-0xc2e0ce8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,761 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,762 DEBUG [hconnection-0x61f5fd94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,765 DEBUG [hconnection-0x4125609a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,766 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,766 DEBUG [hconnection-0x37c9ec58-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,773 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,773 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,774 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43682, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:08,775 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:08,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:08,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:08,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:08,776 DEBUG [hconnection-0x5ee3c059-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:08,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:08,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:08,781 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:08,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:08,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:08,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:08,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:08,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:08,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:08,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:08,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:08,789 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:08,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:08,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:08,790 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:08,790 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:08,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:08,801 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:08,803 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:08,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:08,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362448825, max timestamp=1682362448836Num uniques 859; 2023-04-24 18:54:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:08,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362448825, max timestamp=1682362448836Num uniques 859; 2023-04-24 18:54:08,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362448825, max timestamp=1682362448836Num uniques 859; 2023-04-24 18:54:08,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:08,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:08,864 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248f0801fba8f34cdd84bdeb651727e78b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f0801fba8f34cdd84bdeb651727e78b 2023-04-24 18:54:08,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:08,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362508928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:08,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362448892, max timestamp=1682362448933Num uniques 1018; 2023-04-24 18:54:08,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:08,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362448892, max timestamp=1682362448933Num uniques 1018; 2023-04-24 18:54:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:08,944 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/999c7ca63c20449f84f5ffd3e2f5d58a 2023-04-24 18:54:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362448892, max timestamp=1682362448933Num uniques 1018; 2023-04-24 18:54:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:09,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1682362448965, max timestamp=1682362449018Num uniques 1027; 2023-04-24 18:54:09,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1682362448965, max timestamp=1682362449018Num uniques 1027; 2023-04-24 18:54:09,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:54:09,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682362448965, max timestamp=1682362449018Num uniques 1027; 2023-04-24 18:54:09,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:54:09,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13086, totalHeapSize=29984, min timestamp=1682362449035, max timestamp=1682362449050Num uniques 1086; 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=13086, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13091, totalHeapSize=29984, min timestamp=1682362449035, max timestamp=1682362449050Num uniques 1086; 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13086, totalHeapSize=29984, min timestamp=1682362449035, max timestamp=1682362449050Num uniques 1009; 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49731, new segment data size=13091, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:54:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13086, suffix heap size=132904,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=286 2023-04-24 18:54:09,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362509066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362509067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362509075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362509075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362509176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362509179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362509176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362509179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362509379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362509381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362509381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362509382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,387 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/765734fec0f3484996813d98fa9b9f6b 2023-04-24 18:54:09,456 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/232f0077eaca4943a27339b295215165 2023-04-24 18:54:09,465 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/999c7ca63c20449f84f5ffd3e2f5d58a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a 2023-04-24 18:54:09,471 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a, entries=150, sequenceid=13, filesize=25.1 K 2023-04-24 18:54:09,476 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/765734fec0f3484996813d98fa9b9f6b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b 2023-04-24 18:54:09,492 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b, entries=150, sequenceid=13, filesize=11.6 K 2023-04-24 18:54:09,493 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/232f0077eaca4943a27339b295215165 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165 2023-04-24 18:54:09,498 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165, entries=150, sequenceid=13, filesize=11.6 K 2023-04-24 18:54:09,499 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 103750326d6eb6a8536f7e1c9d4b7a6f in 698ms, sequenceid=13, compaction requested=false 2023-04-24 18:54:09,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:09,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:09,502 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70aae97f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:09,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:09,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,507 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:09,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:09,513 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:09,514 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,514 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:09,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:09,518 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:09,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:09,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:09,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:09,518 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:09,521 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,521 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,521 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:09,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,527 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:09,527 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,627 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:09,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:09,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:09,642 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:09,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:09,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:09,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:09,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=154.31 KB heapSize=433.55 KB 2023-04-24 18:54:09,643 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:09,643 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:09,643 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:09,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:09,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:09,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:09,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:09,648 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:09,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:09,649 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:09,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:09,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:09,653 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:09,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784fda82[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:09,656 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:09,656 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:09,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:09,658 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,658 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:09,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:09,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:09,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:09,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:09,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:09,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:09,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:09,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,669 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,678 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:09,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,682 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:09,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,682 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:09,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362509685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362509685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362509687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362509688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,716 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e1e974f837fa49f9a3098c9ae0c863bf to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e1e974f837fa49f9a3098c9ae0c863bf 2023-04-24 18:54:09,747 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/dbb9616973a84f328f129a2cb359f1a7 2023-04-24 18:54:09,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,782 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:09,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:09,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:09,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:09,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:09,810 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:09,810 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,810 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:09,810 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:09,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:09,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:09,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:09,822 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,822 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:09,822 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:09,822 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:09,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:09,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aec774e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:09,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:09,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:09,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:09,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:09,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:09,843 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:09,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:09,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:09,843 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:09,843 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:09,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,847 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:09,852 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:09,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:09,853 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:09,853 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:09,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,953 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:09,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:09,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:09,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:09,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:09,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:09,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:09,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:09,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:09,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:09,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:09,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:09,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:09,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362509961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:09,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:09,968 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:09,968 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:09,968 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:09,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:09,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:09,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:09,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,979 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:09,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8e19bb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:09,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:09,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:09,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:09,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:09,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:09,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:09,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:09,986 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:09,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:09,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:09,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:09,986 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:09,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:09,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:09,994 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:09,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:10,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,000 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:10,000 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:10,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:10,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,101 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:10,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:10,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:10,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:10,110 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:10,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:10,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:10,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:10,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:10,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:10,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:10,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:10,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:10,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:10,119 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:10,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@848fc24[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:10,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:10,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:10,124 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:10,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:10,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:10,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:10,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:10,129 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:10,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:10,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:10,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:10,129 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:10,131 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,131 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,131 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,132 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:10,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:10,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,136 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,136 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:10,136 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:10,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362510190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362510191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362510191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362510194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ba03a62970564885adce57dd2bae8aef 2023-04-24 18:54:10,237 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:10,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:10,256 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:10,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:10,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:10,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,259 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:10,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362510265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:10,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:10,293 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:10,293 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:10,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:10,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:10,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:10,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:10,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:10,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:10,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed5bc65[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:10,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:10,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:10,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:10,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:10,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:10,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:10,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:10,312 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:10,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:10,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:10,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:10,312 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:10,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,316 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,319 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:10,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,324 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:10,324 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:10,360 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:54:10,424 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:10,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:10,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:10,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:10,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:10,428 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:10,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:10,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:10,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:10,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:10,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:10,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:10,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:10,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:10,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:10,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:10,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:10,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:10,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,439 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,439 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,439 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:10,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67eafcf6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:10,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:10,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,442 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,443 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:10,443 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:10,443 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:10,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:10,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:10,445 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:10,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:10,445 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:10,449 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,449 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,449 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,452 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,458 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,458 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,458 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,459 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:10,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:10,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,459 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:10,459 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,459 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,559 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:10,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:10,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:10,566 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:10,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:10,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:10,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:10,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:10,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:10,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:10,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,577 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:10,577 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:10,577 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:10,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:10,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38766271[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:10,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:10,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:10,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:10,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:10,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:10,588 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,589 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:10,589 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,589 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:10,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:10,600 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:10,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:10,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:10,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:10,600 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:10,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:10,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,606 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:10,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:10,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:10,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:10,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:10,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:10,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,610 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:10,610 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:10,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:10,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/7240535e800b40458bca3a776fcf650c 2023-04-24 18:54:10,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/dbb9616973a84f328f129a2cb359f1a7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7 2023-04-24 18:54:10,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7, entries=200, sequenceid=102, filesize=31.9 K 2023-04-24 18:54:10,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ba03a62970564885adce57dd2bae8aef as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef 2023-04-24 18:54:10,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef, entries=150, sequenceid=102, filesize=11.6 K 2023-04-24 18:54:10,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/7240535e800b40458bca3a776fcf650c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c 2023-04-24 18:54:10,710 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:10,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:10,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:10,715 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:10,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:10,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:10,716 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,716 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:10,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:10,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:10,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:10,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:10,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:10,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c, entries=150, sequenceid=102, filesize=11.6 K 2023-04-24 18:54:10,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~433.50 KB/443904, currentSize=26.84 KB/27480 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1087ms, sequenceid=102, compaction requested=false 2023-04-24 18:54:10,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:10,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:10,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:10,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:10,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:10,735 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:10,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:10,753 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248b0f6b761fc14998bdcd7fd6187ee4df to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b0f6b761fc14998bdcd7fd6187ee4df 2023-04-24 18:54:10,759 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/f110647278dc40fcbcb4cf0d414a45dd 2023-04-24 18:54:10,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:10,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:10,800 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/57ddb3cf4f8148c6a97fd5f7599a855a 2023-04-24 18:54:10,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362450828, max timestamp=1682362450855Num uniques 859; 2023-04-24 18:54:10,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:10,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362450828, max timestamp=1682362450855Num uniques 859; 2023-04-24 18:54:10,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:10,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:10,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362510905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362511010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362511194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362511198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362511201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362511209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362511214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,265 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/0e871729a77b4cfda789a67ecb07fe41 2023-04-24 18:54:11,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/f110647278dc40fcbcb4cf0d414a45dd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd 2023-04-24 18:54:11,278 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd, entries=100, sequenceid=109, filesize=18.4 K 2023-04-24 18:54:11,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/57ddb3cf4f8148c6a97fd5f7599a855a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a 2023-04-24 18:54:11,284 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a, entries=100, sequenceid=109, filesize=9.3 K 2023-04-24 18:54:11,285 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/0e871729a77b4cfda789a67ecb07fe41 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41 2023-04-24 18:54:11,293 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41, entries=100, sequenceid=109, filesize=9.3 K 2023-04-24 18:54:11,295 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=152.07 KB/155720 for 103750326d6eb6a8536f7e1c9d4b7a6f in 560ms, sequenceid=109, compaction requested=true 2023-04-24 18:54:11,295 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:11,295 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:11,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:11,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:11,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:11,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:11,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,298 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:11,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77101087[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:11,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:11,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:11,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:11,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:11,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:11,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:11,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:11,307 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:11,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:11,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:11,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:11,307 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:11,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,309 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:11,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:11,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:11,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:11,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:11,327 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:11,328 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:11,428 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:11,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:11,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:11,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:11,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:11,436 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:11,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:11,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:11,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:11,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:11,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:11,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:11,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:11,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:11,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:11,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:11,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:11,443 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:11,443 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:11,443 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:11,443 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:11,443 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=152.07 KB heapSize=443.08 KB 2023-04-24 18:54:11,443 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:11,443 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:11,443 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:11,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:11,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:11,444 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:11,472 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424f73c2aa0b76246dfaa86d9d8e5c12769 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f73c2aa0b76246dfaa86d9d8e5c12769 2023-04-24 18:54:11,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:11,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362511531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362511635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:11,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43638 deadline: 1682362511838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,883 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a2632c43c40145049882599d98d1673f 2023-04-24 18:54:11,895 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/7dede12f8d4a429280f0e9d10c5ab879 2023-04-24 18:54:11,909 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d9f690f758d540b4b6fe9cdff565cca5 2023-04-24 18:54:11,917 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a2632c43c40145049882599d98d1673f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f 2023-04-24 18:54:11,922 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f, entries=150, sequenceid=144, filesize=25.3 K 2023-04-24 18:54:11,925 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/7dede12f8d4a429280f0e9d10c5ab879 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879 2023-04-24 18:54:11,929 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879, entries=150, sequenceid=144, filesize=11.8 K 2023-04-24 18:54:11,930 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d9f690f758d540b4b6fe9cdff565cca5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5 2023-04-24 18:54:11,966 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5, entries=150, sequenceid=144, filesize=11.8 K 2023-04-24 18:54:11,967 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~443.03 KB/453664, currentSize=26.84 KB/27480 for 103750326d6eb6a8536f7e1c9d4b7a6f in 524ms, sequenceid=144, compaction requested=true 2023-04-24 18:54:11,967 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:11,967 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:11,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:11,974 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5405ac94[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:11,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:11,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,978 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,978 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:11,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:11,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:11,981 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:11,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:11,985 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:11,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:11,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:11,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:11,985 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:11,987 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,987 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:11,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,987 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:11,987 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,990 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:11,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:11,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:11,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:11,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:11,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:11,993 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:11,994 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:11,994 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:12,094 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:12,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:12,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:12,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:12,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:12,099 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:12,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:12,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:12,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,105 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:12,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:12,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:12,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:12,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:12,107 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,107 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:12,107 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:12,135 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a9bb756207334a5db84b0f9e5478731f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a9bb756207334a5db84b0f9e5478731f 2023-04-24 18:54:12,146 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/367c80d31ff24caa90cf617e55d917e8 2023-04-24 18:54:12,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:12,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:12,183 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/4da6256a54264f37907c07497d8fbeeb 2023-04-24 18:54:12,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362452184, max timestamp=1682362452266Num uniques 859; 2023-04-24 18:54:12,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:54:12,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362452184, max timestamp=1682362452266Num uniques 859; 2023-04-24 18:54:12,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:54:12,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362452184, max timestamp=1682362452266Num uniques 859; 2023-04-24 18:54:12,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:54:12,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362452298, max timestamp=1682362452328Num uniques 1068; 2023-04-24 18:54:12,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:54:12,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362452298, max timestamp=1682362452328Num uniques 1068; 2023-04-24 18:54:12,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362452298, max timestamp=1682362452328Num uniques 1068; 2023-04-24 18:54:12,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:54:12,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:54:12,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362452377, max timestamp=1682362452393Num uniques 1127; 2023-04-24 18:54:12,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362452377, max timestamp=1682362452393Num uniques 1127; 2023-04-24 18:54:12,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:54:12,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:54:12,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362452377, max timestamp=1682362452393Num uniques 1127; 2023-04-24 18:54:12,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:54:12,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362452428, max timestamp=1682362452452Num uniques 1036; 2023-04-24 18:54:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 18:54:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362452428, max timestamp=1682362452452Num uniques 1036; 2023-04-24 18:54:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 18:54:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362452428, max timestamp=1682362452452Num uniques 1036; 2023-04-24 18:54:12,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 18:54:12,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362452495, max timestamp=1682362452509Num uniques 1145; 2023-04-24 18:54:12,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 18:54:12,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362452495, max timestamp=1682362452509Num uniques 1145; 2023-04-24 18:54:12,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 18:54:12,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362452495, max timestamp=1682362452509Num uniques 1145; 2023-04-24 18:54:12,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 18:54:12,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362452548, max timestamp=1682362452578Num uniques 1104; 2023-04-24 18:54:12,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:12,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362452548, max timestamp=1682362452578Num uniques 1104; 2023-04-24 18:54:12,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362452548, max timestamp=1682362452578Num uniques 1104; 2023-04-24 18:54:12,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:12,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:12,607 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/78c7a0c1cf4248dd8e9391bba956aa65 2023-04-24 18:54:12,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/367c80d31ff24caa90cf617e55d917e8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8 2023-04-24 18:54:12,617 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8, entries=100, sequenceid=151, filesize=18.5 K 2023-04-24 18:54:12,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/4da6256a54264f37907c07497d8fbeeb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb 2023-04-24 18:54:12,622 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb, entries=100, sequenceid=151, filesize=9.4 K 2023-04-24 18:54:12,623 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/78c7a0c1cf4248dd8e9391bba956aa65 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65 2023-04-24 18:54:12,629 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65, entries=100, sequenceid=151, filesize=9.4 K 2023-04-24 18:54:12,630 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=140.89 KB/144270 for 103750326d6eb6a8536f7e1c9d4b7a6f in 523ms, sequenceid=151, compaction requested=true 2023-04-24 18:54:12,630 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:12,630 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,632 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,633 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,633 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:12,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b8f7ae[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:12,635 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:12,635 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:12,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:12,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:12,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:12,641 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:12,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:12,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:12,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:12,641 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:12,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:12,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=147.60 KB heapSize=421.31 KB 2023-04-24 18:54:12,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:12,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:12,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:12,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:12,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:12,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:12,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:12,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:12,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,652 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:12,652 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:12,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:12,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362512669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,752 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:12,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:12,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:12,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:12,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:12,756 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:12,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:12,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:12,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,758 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,758 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,758 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,758 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:12,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:12,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:12,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:12,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:12,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:12,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:12,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:12,762 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5674a2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:12,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:12,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,771 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:12,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:12,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:12,771 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:12,773 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:12,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:12,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362512773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:12,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:12,776 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:12,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:12,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:12,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:12,776 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:12,778 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,778 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,778 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,778 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,782 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:12,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:12,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,783 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,783 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,783 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,783 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,783 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:12,783 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:12,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,883 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:12,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:12,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:12,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:12,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:12,886 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:12,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:12,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:12,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:12,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:12,891 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,891 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:12,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:12,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:12,893 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b5d5e7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:12,895 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:12,895 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:12,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:12,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:12,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:12,900 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:12,900 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:12,900 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:12,900 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:12,900 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:12,903 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,903 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,903 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:12,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:12,907 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:12,907 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:12,907 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:12,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:12,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:12,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:12,908 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:12,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:12,908 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:12,908 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:12,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362512976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,008 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,013 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,015 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612dab69[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,025 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,025 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,026 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,028 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,031 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,031 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,033 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,033 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,033 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,038 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,038 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,060 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042449f49b7e41dc41ddbeab6978f7383bcd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042449f49b7e41dc41ddbeab6978f7383bcd 2023-04-24 18:54:13,139 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,142 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,143 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,143 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,147 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,147 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,147 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,147 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,149 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428b2565[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,162 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,164 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,164 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,164 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,164 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,165 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,165 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362513202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,207 DEBUG [Thread-852] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:13,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362513207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,211 DEBUG [Thread-854] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:13,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362513215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,220 DEBUG [Thread-856] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:13,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362513221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,223 DEBUG [Thread-850] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:13,265 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,268 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,268 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,268 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,268 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,269 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,269 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,271 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,273 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,273 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,273 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e24aaf2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,279 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,281 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362513280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,285 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,285 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,287 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,287 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,287 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,288 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,293 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,293 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,393 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,396 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,398 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,400 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,400 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,400 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,400 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28084e3c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,405 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,406 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,416 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,418 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,419 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,422 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,426 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,426 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,466 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/10b462566a414b0198a865fbdd7b137a 2023-04-24 18:54:13,526 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,530 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,531 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,532 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,538 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,538 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,538 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,538 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2472ceb7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,543 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,546 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,549 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,550 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,564 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,565 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,565 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,665 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,669 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,676 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,676 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,677 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f497751[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,684 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,685 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,686 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,686 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,686 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,695 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,695 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362513787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,795 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,799 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,799 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,799 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,799 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,804 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4813970[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,809 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,809 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,810 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,816 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,816 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,824 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,833 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,833 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/5c898b0a89d346d6bf29f5a91fa9acad 2023-04-24 18:54:13,933 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:13,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:13,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:13,937 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:13,937 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:13,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:13,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,940 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:13,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:13,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:13,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:13,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:13,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:13,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e12fe72[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:13,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:13,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:13,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:13,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:13,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:13,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:13,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:13,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:13,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:13,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:13,956 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:13,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,961 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,961 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:13,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:13,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:13,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:13,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:13,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:13,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:13,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:13,963 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:13,963 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:13,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,063 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:14,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:14,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:14,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:14,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:14,072 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:14,072 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,072 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:14,072 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:14,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:14,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:14,076 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:14,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:14,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:14,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:14,079 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c278271[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:14,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:14,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,082 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:14,082 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,082 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:14,083 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,083 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:14,083 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,083 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:14,086 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:14,086 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:14,086 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:14,086 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:14,086 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:14,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,092 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:14,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:14,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,092 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:14,092 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:14,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,193 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:14,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:14,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:14,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:14,196 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:14,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:14,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:14,197 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,197 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,198 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:14,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:14,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:14,199 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:14,199 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:14,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:14,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:14,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:14,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a0855b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:14,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:14,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:14,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:14,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:14,210 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:14,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:14,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:14,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:14,211 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:14,212 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,212 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:14,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,212 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:14,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:14,213 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:14,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:14,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,219 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,220 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:14,220 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:14,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:14,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d9b2b2c8db574b3489e6e3f40f88330f 2023-04-24 18:54:14,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/10b462566a414b0198a865fbdd7b137a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a 2023-04-24 18:54:14,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a, entries=200, sequenceid=275, filesize=32.3 K 2023-04-24 18:54:14,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/5c898b0a89d346d6bf29f5a91fa9acad as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad 2023-04-24 18:54:14,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad, entries=150, sequenceid=275, filesize=11.9 K 2023-04-24 18:54:14,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d9b2b2c8db574b3489e6e3f40f88330f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f 2023-04-24 18:54:14,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f, entries=150, sequenceid=275, filesize=11.9 K 2023-04-24 18:54:14,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~421.27 KB/431376, currentSize=33.54 KB/34350 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1674ms, sequenceid=275, compaction requested=true 2023-04-24 18:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:14,319 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:54:14,319 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:54:14,320 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:14,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:14,322 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155110 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:54:14,322 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67178 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:54:14,322 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:14,322 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:14,322 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,322 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,322 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=151.5 K 2023-04-24 18:54:14,322 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=65.6 K 2023-04-24 18:54:14,323 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 765734fec0f3484996813d98fa9b9f6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682362448781 2023-04-24 18:54:14,323 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 999c7ca63c20449f84f5ffd3e2f5d58a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682362448781 2023-04-24 18:54:14,324 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ba03a62970564885adce57dd2bae8aef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682362449069 2023-04-24 18:54:14,324 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting dbb9616973a84f328f129a2cb359f1a7, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682362449059 2023-04-24 18:54:14,324 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 57ddb3cf4f8148c6a97fd5f7599a855a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362449652 2023-04-24 18:54:14,324 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f110647278dc40fcbcb4cf0d414a45dd, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362449652 2023-04-24 18:54:14,325 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7dede12f8d4a429280f0e9d10c5ab879, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682362450899 2023-04-24 18:54:14,325 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a2632c43c40145049882599d98d1673f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682362450899 2023-04-24 18:54:14,325 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4da6256a54264f37907c07497d8fbeeb, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362451526 2023-04-24 18:54:14,325 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 367c80d31ff24caa90cf617e55d917e8, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362451526 2023-04-24 18:54:14,326 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5c898b0a89d346d6bf29f5a91fa9acad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:14,326 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 10b462566a414b0198a865fbdd7b137a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:14,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:14,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:14,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:14,327 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:14,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:14,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:14,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,332 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:14,332 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:14,332 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:14,332 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:14,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:14,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:14,334 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,334 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:14,334 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:54:14,334 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:14,335 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:14,335 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:14,335 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:14,335 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:14,335 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:14,361 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#134 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:14,382 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#133 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:14,398 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/fe933f3d0da34c6193df8301d8d400f9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/fe933f3d0da34c6193df8301d8d400f9 2023-04-24 18:54:14,400 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304247320ff37583e4a26ae3e2fecf6f07aed to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304247320ff37583e4a26ae3e2fecf6f07aed 2023-04-24 18:54:14,406 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into fe933f3d0da34c6193df8301d8d400f9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:14,406 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:14,406 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=10, startTime=1682362454319; duration=0sec 2023-04-24 18:54:14,406 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:14,406 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:54:14,408 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67178 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:54:14,408 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:14,408 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:14,408 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=65.6 K 2023-04-24 18:54:14,409 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 232f0077eaca4943a27339b295215165, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682362448781 2023-04-24 18:54:14,409 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7240535e800b40458bca3a776fcf650c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682362449069 2023-04-24 18:54:14,409 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0e871729a77b4cfda789a67ecb07fe41, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362449652 2023-04-24 18:54:14,411 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d9f690f758d540b4b6fe9cdff565cca5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682362450899 2023-04-24 18:54:14,412 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 78c7a0c1cf4248dd8e9391bba956aa65, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362451526 2023-04-24 18:54:14,412 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d9b2b2c8db574b3489e6e3f40f88330f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:14,426 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#135 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:14,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:14,812 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/efca336650b4406db10c1ec3a26983ba 2023-04-24 18:54:14,829 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/881f0de6e42f49769acd0db1d7da598d 2023-04-24 18:54:14,835 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/3f36f4e911f34eeda749495c5906c746 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/3f36f4e911f34eeda749495c5906c746 2023-04-24 18:54:14,839 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/8fdfecbf00ba44c0ac725b741c16feec as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fdfecbf00ba44c0ac725b741c16feec 2023-04-24 18:54:14,842 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into 3f36f4e911f34eeda749495c5906c746(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:14,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:14,842 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=10, startTime=1682362454319; duration=0sec 2023-04-24 18:54:14,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:14,845 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 8fdfecbf00ba44c0ac725b741c16feec(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:14,845 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:14,845 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=10, startTime=1682362454319; duration=0sec 2023-04-24 18:54:14,845 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:14,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362454823, max timestamp=1682362454863Num uniques 859; 2023-04-24 18:54:14,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:14,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362454823, max timestamp=1682362454863Num uniques 859; 2023-04-24 18:54:14,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362454823, max timestamp=1682362454863Num uniques 859; 2023-04-24 18:54:14,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:14,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:14,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362454888, max timestamp=1682362454909Num uniques 1018; 2023-04-24 18:54:14,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:14,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362454888, max timestamp=1682362454909Num uniques 1018; 2023-04-24 18:54:14,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:14,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362454888, max timestamp=1682362454909Num uniques 1018; 2023-04-24 18:54:14,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:14,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362454945, max timestamp=1682362454957Num uniques 1077; 2023-04-24 18:54:14,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362454945, max timestamp=1682362454957Num uniques 1077; 2023-04-24 18:54:14,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:14,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:14,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362454945, max timestamp=1682362454957Num uniques 1077; 2023-04-24 18:54:14,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:14,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:14,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362514993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362515095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,251 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/54758161cd124cf29466ce1ad2454f4a 2023-04-24 18:54:15,257 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/efca336650b4406db10c1ec3a26983ba as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba 2023-04-24 18:54:15,261 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba, entries=150, sequenceid=283, filesize=25.4 K 2023-04-24 18:54:15,261 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/881f0de6e42f49769acd0db1d7da598d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d 2023-04-24 18:54:15,266 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d, entries=150, sequenceid=283, filesize=11.9 K 2023-04-24 18:54:15,267 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/54758161cd124cf29466ce1ad2454f4a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a 2023-04-24 18:54:15,271 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a, entries=150, sequenceid=283, filesize=11.9 K 2023-04-24 18:54:15,272 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 103750326d6eb6a8536f7e1c9d4b7a6f in 938ms, sequenceid=283, compaction requested=false 2023-04-24 18:54:15,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:15,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25677386[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,287 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,287 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,294 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,294 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:15,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:15,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=154.31 KB heapSize=434.39 KB 2023-04-24 18:54:15,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:15,300 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:15,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:15,300 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:15,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:15,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:15,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:15,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362515316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,394 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:15,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:15,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:15,398 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:15,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:15,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:15,400 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,400 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,401 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:15,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7fe836[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,410 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,413 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,413 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,415 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,420 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,420 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:15,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:15,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362515420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,520 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:15,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:15,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:15,525 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:15,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:15,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:15,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:15,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:15,531 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,531 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:15,531 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:15,531 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,532 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2bddc2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,556 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,557 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,568 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:15,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,568 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:15,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:15,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362515623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,668 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:15,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:15,688 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:15,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:15,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:15,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,690 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:15,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:15,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:15,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:15,693 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,693 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:15,693 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:15,693 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108d0d41[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,703 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,708 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,708 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,717 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,719 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042426690001db344663a73802c609be35e1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042426690001db344663a73802c609be35e1 2023-04-24 18:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,724 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,724 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:15,727 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/c949189839a14255897460cfd017e6b2 2023-04-24 18:54:15,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/51f89a7dc3d64f5c8a6f318b462d1909 2023-04-24 18:54:15,824 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:15,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:15,827 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:15,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:15,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:15,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:15,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:15,831 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,831 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:15,831 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:15,831 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee8b795[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,846 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,846 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,847 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,847 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,847 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,854 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:15,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,854 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,854 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362515929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,954 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:15,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:15,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:15,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:15,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:15,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:15,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:15,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:15,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:15,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:15,962 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,962 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:15,962 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:15,962 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:15,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:15,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:15,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:15,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:15,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,966 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:15,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d75c026[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:15,968 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:15,968 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:15,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:15,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:15,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:15,972 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:15,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:15,974 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:15,974 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:15,974 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:15,974 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:15,974 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:15,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:15,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:15,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,978 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:15,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:15,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:15,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:15,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:15,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:15,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:15,985 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:15,985 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:16,085 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:16,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:16,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:16,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:16,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:16,089 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:16,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:16,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:16,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,091 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:16,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:16,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:16,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:16,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:16,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:16,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:16,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:16,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,097 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:16,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd2cd87[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:16,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:16,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,100 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:16,101 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,101 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:16,102 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:16,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:16,105 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:16,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:16,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:16,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:16,105 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:16,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,108 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:16,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:16,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,113 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:16,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:16,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,113 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:16,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:16,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/17c78fdfafff4fd288784c5d81cc27cc 2023-04-24 18:54:16,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/c949189839a14255897460cfd017e6b2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2 2023-04-24 18:54:16,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2, entries=150, sequenceid=358, filesize=25.4 K 2023-04-24 18:54:16,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/51f89a7dc3d64f5c8a6f318b462d1909 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909 2023-04-24 18:54:16,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909, entries=150, sequenceid=358, filesize=11.9 K 2023-04-24 18:54:16,213 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:16,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:16,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:16,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:16,216 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:16,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:16,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:16,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,218 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:16,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:16,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:16,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:16,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/17c78fdfafff4fd288784c5d81cc27cc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc 2023-04-24 18:54:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:16,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:16,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:16,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:16,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:16,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:16,221 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:16,221 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,221 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:16,221 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:16,222 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:16,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374d2639[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:16,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:16,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:16,232 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:16,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc, entries=150, sequenceid=358, filesize=11.9 K 2023-04-24 18:54:16,233 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:16,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 103750326d6eb6a8536f7e1c9d4b7a6f in 934ms, sequenceid=358, compaction requested=true 2023-04-24 18:54:16,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:16,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:16,234 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:16,234 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:16,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:16,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,236 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78344 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:16,236 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:16,236 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,236 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/3f36f4e911f34eeda749495c5906c746, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=76.5 K 2023-04-24 18:54:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,236 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:16,236 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:16,236 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,236 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/fe933f3d0da34c6193df8301d8d400f9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=35.9 K 2023-04-24 18:54:16,237 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3f36f4e911f34eeda749495c5906c746, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:16,237 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting efca336650b4406db10c1ec3a26983ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682362452654 2023-04-24 18:54:16,237 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fe933f3d0da34c6193df8301d8d400f9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:16,237 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c949189839a14255897460cfd017e6b2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:16,238 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 881f0de6e42f49769acd0db1d7da598d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682362452654 2023-04-24 18:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,238 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 51f89a7dc3d64f5c8a6f318b462d1909, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:16,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:16,239 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:16,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:16,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:16,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:16,240 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:16,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:16,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:16,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:16,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:16,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:16,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:16,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:16,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:16,251 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:16,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,251 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:16,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:16,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:16,253 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:16,256 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#142 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:16,351 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:16,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:16,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:16,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:16,355 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:16,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:16,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:16,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,357 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:16,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:16,358 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:16,358 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:16,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:16,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:16,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:16,359 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,359 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:16,359 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:16,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:16,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362456464, max timestamp=1682362456479Num uniques 859; 2023-04-24 18:54:16,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362456464, max timestamp=1682362456479Num uniques 859; 2023-04-24 18:54:16,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362456464, max timestamp=1682362456479Num uniques 859; 2023-04-24 18:54:16,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:16,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:16,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:16,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362456507, max timestamp=1682362456530Num uniques 1018; 2023-04-24 18:54:16,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:16,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362456507, max timestamp=1682362456530Num uniques 1018; 2023-04-24 18:54:16,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:16,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362456507, max timestamp=1682362456530Num uniques 1018; 2023-04-24 18:54:16,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:16,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362456549, max timestamp=1682362456583Num uniques 1077; 2023-04-24 18:54:16,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:16,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362456549, max timestamp=1682362456583Num uniques 1077; 2023-04-24 18:54:16,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362456549, max timestamp=1682362456583Num uniques 1077; 2023-04-24 18:54:16,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:16,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-24 18:54:16,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362456628, max timestamp=1682362456640Num uniques 1136; 2023-04-24 18:54:16,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-24 18:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362456628, max timestamp=1682362456640Num uniques 1136; 2023-04-24 18:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362456628, max timestamp=1682362456640Num uniques 1136; 2023-04-24 18:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-24 18:54:16,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-24 18:54:16,667 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/73f018eb946e4764acdb355c127c2c95 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/73f018eb946e4764acdb355c127c2c95 2023-04-24 18:54:16,672 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into 73f018eb946e4764acdb355c127c2c95(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:16,673 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:16,673 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362456234; duration=0sec 2023-04-24 18:54:16,673 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:16,673 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:16,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:16,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:16,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:16,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fdfecbf00ba44c0ac725b741c16feec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=35.9 K 2023-04-24 18:54:16,675 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8fdfecbf00ba44c0ac725b741c16feec, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1682362452616 2023-04-24 18:54:16,675 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 54758161cd124cf29466ce1ad2454f4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682362452654 2023-04-24 18:54:16,676 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 17c78fdfafff4fd288784c5d81cc27cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:16,683 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:16,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362456693, max timestamp=1682362456709Num uniques 1045; 2023-04-24 18:54:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-24 18:54:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362456693, max timestamp=1682362456709Num uniques 1045; 2023-04-24 18:54:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-24 18:54:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362456693, max timestamp=1682362456709Num uniques 1045; 2023-04-24 18:54:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362456721, max timestamp=1682362456761Num uniques 1104; 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362456721, max timestamp=1682362456761Num uniques 1104; 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362456721, max timestamp=1682362456761Num uniques 1104; 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 18:54:16,779 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a3050f03abe74c3a98316081b5e77160 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a3050f03abe74c3a98316081b5e77160 2023-04-24 18:54:16,784 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=365, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/5b298951cb474ee5b628d38410e8cfb4 2023-04-24 18:54:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682362456797, max timestamp=1682362456818Num uniques 1063; 2023-04-24 18:54:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-24 18:54:16,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682362456797, max timestamp=1682362456818Num uniques 1063; 2023-04-24 18:54:16,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682362456797, max timestamp=1682362456818Num uniques 1063; 2023-04-24 18:54:16,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-24 18:54:16,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362456858, max timestamp=1682362456869Num uniques 1072; 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362456858, max timestamp=1682362456869Num uniques 1072; 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362456858, max timestamp=1682362456869Num uniques 1072; 2023-04-24 18:54:16,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362456889, max timestamp=1682362456909Num uniques 1031; 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362456889, max timestamp=1682362456909Num uniques 1031; 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362456889, max timestamp=1682362456909Num uniques 1031; 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 18:54:16,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 18:54:16,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682362456947, max timestamp=1682362456964Num uniques 1090; 2023-04-24 18:54:16,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 18:54:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682362456947, max timestamp=1682362456964Num uniques 1090; 2023-04-24 18:54:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 18:54:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682362456947, max timestamp=1682362456964Num uniques 1090; 2023-04-24 18:54:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 18:54:17,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682362456993, max timestamp=1682362457036Num uniques 1099; 2023-04-24 18:54:17,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-24 18:54:17,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682362456993, max timestamp=1682362457036Num uniques 1099; 2023-04-24 18:54:17,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-24 18:54:17,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682362456993, max timestamp=1682362457036Num uniques 1099; 2023-04-24 18:54:17,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-24 18:54:17,079 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a8079503481f4f6d849d9b902063b8f3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a8079503481f4f6d849d9b902063b8f3 2023-04-24 18:54:17,084 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into a8079503481f4f6d849d9b902063b8f3(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:17,084 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:17,084 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362456234; duration=0sec 2023-04-24 18:54:17,084 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:17,091 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/7ec3f389991f4dc5af4e3459ca53160a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7ec3f389991f4dc5af4e3459ca53160a 2023-04-24 18:54:17,096 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 7ec3f389991f4dc5af4e3459ca53160a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:17,096 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:17,096 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362456234; duration=0sec 2023-04-24 18:54:17,096 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:17,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682362457085, max timestamp=1682362457100Num uniques 1108; 2023-04-24 18:54:17,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 18:54:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682362457085, max timestamp=1682362457100Num uniques 1108; 2023-04-24 18:54:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 18:54:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682362457085, max timestamp=1682362457100Num uniques 1108; 2023-04-24 18:54:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 18:54:17,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1682362457119, max timestamp=1682362457143Num uniques 1067; 2023-04-24 18:54:17,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-24 18:54:17,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1682362457119, max timestamp=1682362457143Num uniques 1067; 2023-04-24 18:54:17,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-24 18:54:17,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682362457134, max timestamp=1682362457143Num uniques 1067; 2023-04-24 18:54:17,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-24 18:54:17,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682362457184, max timestamp=1682362457193Num uniques 1176; 2023-04-24 18:54:17,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=8056, suffix heap size=150272,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=176 2023-04-24 18:54:17,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682362457184, max timestamp=1682362457193Num uniques 1176; 2023-04-24 18:54:17,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=8056, suffix heap size=150272,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=176 2023-04-24 18:54:17,196 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ac0f99035e944164943a7b4c50c9ad91 2023-04-24 18:54:17,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682362457184, max timestamp=1682362457193Num uniques 1026; 2023-04-24 18:54:17,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-24 18:54:17,215 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/fc66117c1f2648d99e73ae2b93582e46 2023-04-24 18:54:17,221 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/5b298951cb474ee5b628d38410e8cfb4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4 2023-04-24 18:54:17,227 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4, entries=150, sequenceid=365, filesize=25.4 K 2023-04-24 18:54:17,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ac0f99035e944164943a7b4c50c9ad91 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91 2023-04-24 18:54:17,232 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91, entries=150, sequenceid=365, filesize=11.9 K 2023-04-24 18:54:17,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/fc66117c1f2648d99e73ae2b93582e46 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46 2023-04-24 18:54:17,237 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46, entries=150, sequenceid=365, filesize=11.9 K 2023-04-24 18:54:17,238 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for 103750326d6eb6a8536f7e1c9d4b7a6f in 879ms, sequenceid=365, compaction requested=false 2023-04-24 18:54:17,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:17,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:17,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=107.34 KB heapSize=303.70 KB 2023-04-24 18:54:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd26b5a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:17,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:17,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,258 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,259 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362517255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362517258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,264 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 558 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362517259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362517259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362517260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,271 DEBUG [Thread-850] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:17,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,277 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,277 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,279 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a64bd7b4a0104c6e9c8774ec936887b1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a64bd7b4a0104c6e9c8774ec936887b1 2023-04-24 18:54:17,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362517360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362517363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 560 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362517366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362517366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,377 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:17,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:17,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:17,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:17,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:17,380 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:17,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:17,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:17,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:17,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:17,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:17,384 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,384 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:17,384 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:17,384 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,386 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f25f016[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,390 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,390 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,394 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,397 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,397 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,398 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,398 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,398 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,400 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,403 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,403 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,504 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:17,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:17,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:17,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:17,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:17,507 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:17,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:17,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:17,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:17,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:17,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,511 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,513 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,514 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2642b480[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,515 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,515 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,516 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,517 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,517 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,520 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,520 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,520 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,520 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,520 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,523 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,523 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,523 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:17,527 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,528 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362517564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362517567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 562 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362517569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362517570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,628 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:17,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:17,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:17,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:17,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:17,631 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:17,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:17,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:17,632 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,632 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:17,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:17,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:17,634 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,635 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:17,635 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:17,635 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7860329f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,638 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,638 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,645 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,645 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,648 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,652 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,652 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,692 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=625, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a1be131db8c14474ad1422e411e195a9 2023-04-24 18:54:17,752 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:17,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:17,756 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:17,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:17,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:17,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,758 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,758 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,759 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,759 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:17,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:17,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:17,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:17,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,762 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17871e72[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,766 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,766 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,766 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,766 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,766 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,766 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,767 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,770 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,770 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,781 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,782 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:17,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,785 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,785 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,785 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362517870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362517872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 564 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362517872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:17,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362517874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,886 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:17,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:17,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:17,889 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,891 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,891 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:17,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:17,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:17,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:17,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:17,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:17,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:17,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:17,893 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,893 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:17,893 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:17,893 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:17,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:17,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:17,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:17,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:17,895 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,895 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:17,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4772e9d4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:17,897 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:17,897 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:17,899 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,899 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:17,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:17,903 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:17,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:17,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:17,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:17,903 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:17,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:17,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:17,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:17,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:17,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:17,910 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:17,910 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:17,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:17,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:17,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,010 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:18,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:18,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:18,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:18,014 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:18,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:18,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:18,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,015 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:18,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:18,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:18,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:18,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:18,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:18,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:18,019 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:18,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:18,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7427a30e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:18,022 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:18,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:18,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:18,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:18,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:18,028 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:18,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:18,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:18,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:18,028 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:18,030 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,030 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,030 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,032 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,035 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:18,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:18,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,036 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:18,036 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:18,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/f6ed334068b44d2895350a1c6c10d274 2023-04-24 18:54:18,136 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:18,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:18,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:18,141 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:18,141 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,141 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:18,141 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:18,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,143 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,143 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:18,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:18,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:18,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:18,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:18,146 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,146 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:18,146 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:18,146 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:18,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d2a93df[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:18,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:18,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:18,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:18,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,157 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:18,163 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:18,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:18,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:18,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:18,163 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,168 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,175 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:18,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:18,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:18,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,175 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:18,175 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:18,276 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:18,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:18,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:18,279 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:18,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:18,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:18,280 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,280 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,281 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:18,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:18,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:18,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:18,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:18,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f17886[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:18,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:18,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,295 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:18,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:18,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:18,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:18,299 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:18,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:18,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:18,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:18,299 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:18,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,303 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:18,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:18,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:18,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,307 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:18,307 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:18,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362518375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:18,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 566 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362518377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:18,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362518377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:18,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362518379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,407 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:18,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:18,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:18,410 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:18,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:18,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:18,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:18,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:18,416 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:18,421 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729d0945[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:18,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:18,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:18,424 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,424 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:18,429 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:18,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:18,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:18,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:18,429 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:18,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:18,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:18,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:18,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:18,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:18,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:18,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:18,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,438 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:18,438 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:18,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:18,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/7956d35ecc064fd49c7ef4e164a57cd4 2023-04-24 18:54:18,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a1be131db8c14474ad1422e411e195a9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9 2023-04-24 18:54:18,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9, entries=250, sequenceid=625, filesize=39.2 K 2023-04-24 18:54:18,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/f6ed334068b44d2895350a1c6c10d274 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274 2023-04-24 18:54:18,538 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:18,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274, entries=150, sequenceid=625, filesize=11.9 K 2023-04-24 18:54:18,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:18,542 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:18,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:18,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:18,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/7956d35ecc064fd49c7ef4e164a57cd4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4 2023-04-24 18:54:18,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:18,545 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:18,545 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:18,545 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:18,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:18,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:18,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:18,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:18,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:18,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:18,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4, entries=150, sequenceid=625, filesize=11.9 K 2023-04-24 18:54:18,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~323.58 KB/331344, currentSize=73.80 KB/75570 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1308ms, sequenceid=625, compaction requested=true 2023-04-24 18:54:18,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:18,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:18,552 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:18,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:18,552 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:18,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:18,553 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92530 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:18,553 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:18,553 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:18,553 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:18,553 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,553 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,554 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a8079503481f4f6d849d9b902063b8f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=90.4 K 2023-04-24 18:54:18,554 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/73f018eb946e4764acdb355c127c2c95, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.0 K 2023-04-24 18:54:18,556 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a8079503481f4f6d849d9b902063b8f3, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:18,556 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 73f018eb946e4764acdb355c127c2c95, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:18,556 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5b298951cb474ee5b628d38410e8cfb4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682362455302 2023-04-24 18:54:18,556 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ac0f99035e944164943a7b4c50c9ad91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682362455302 2023-04-24 18:54:18,556 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a1be131db8c14474ad1422e411e195a9, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:18,557 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f6ed334068b44d2895350a1c6c10d274, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:18,560 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:18,561 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:18,573 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#151 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:18,583 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#150 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:18,997 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/d68cd23f57c24fa6b777f751a0694485 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/d68cd23f57c24fa6b777f751a0694485 2023-04-24 18:54:19,000 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042498a61fe673f64254a94e8ceb9c91286a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498a61fe673f64254a94e8ceb9c91286a 2023-04-24 18:54:19,011 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into d68cd23f57c24fa6b777f751a0694485(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:19,011 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,011 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362458552; duration=0sec 2023-04-24 18:54:19,011 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:19,011 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:19,013 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:19,013 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:19,013 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,014 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7ec3f389991f4dc5af4e3459ca53160a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.0 K 2023-04-24 18:54:19,014 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7ec3f389991f4dc5af4e3459ca53160a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682362454988 2023-04-24 18:54:19,015 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fc66117c1f2648d99e73ae2b93582e46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1682362455302 2023-04-24 18:54:19,015 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7956d35ecc064fd49c7ef4e164a57cd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:19,016 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=639, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/84956f71e616468999e4d0ca3332909f 2023-04-24 18:54:19,029 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#154 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:19,058 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ad0f0023a5fd4710b40f29ce1433854a 2023-04-24 18:54:19,071 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/6b913cb745ba446aac097bcaaf91a250 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b913cb745ba446aac097bcaaf91a250 2023-04-24 18:54:19,076 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/058b63bc5a954aab83ff3baafb1c9643 2023-04-24 18:54:19,083 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into 6b913cb745ba446aac097bcaaf91a250(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:19,083 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,083 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362458552; duration=0sec 2023-04-24 18:54:19,083 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:19,085 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/036367661a3f4f4e9aae48fdaafe77f6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/036367661a3f4f4e9aae48fdaafe77f6 2023-04-24 18:54:19,085 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/84956f71e616468999e4d0ca3332909f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f 2023-04-24 18:54:19,091 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f, entries=150, sequenceid=639, filesize=25.4 K 2023-04-24 18:54:19,091 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 036367661a3f4f4e9aae48fdaafe77f6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:19,092 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,092 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362458552; duration=0sec 2023-04-24 18:54:19,092 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:19,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/ad0f0023a5fd4710b40f29ce1433854a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a 2023-04-24 18:54:19,096 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a, entries=150, sequenceid=639, filesize=11.9 K 2023-04-24 18:54:19,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/058b63bc5a954aab83ff3baafb1c9643 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643 2023-04-24 18:54:19,102 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643, entries=150, sequenceid=639, filesize=11.9 K 2023-04-24 18:54:19,103 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 103750326d6eb6a8536f7e1c9d4b7a6f in 542ms, sequenceid=639, compaction requested=false 2023-04-24 18:54:19,103 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,103 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,105 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bebebcb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,107 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,107 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,109 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,110 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,110 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,117 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,125 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,125 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,225 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,229 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,231 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62863302[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,239 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,245 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,245 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,252 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,252 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,352 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,356 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1a529c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,366 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,366 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,368 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,368 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,368 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,370 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,374 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,374 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,383 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,383 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:19,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:19,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:19,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362519441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682362459422, max timestamp=1682362459444Num uniques 859; 2023-04-24 18:54:19,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682362459422, max timestamp=1682362459444Num uniques 859; 2023-04-24 18:54:19,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 574 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362519442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682362459422, max timestamp=1682362459444Num uniques 859; 2023-04-24 18:54:19,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-24 18:54:19,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-24 18:54:19,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-24 18:54:19,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362519449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362519453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,483 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,486 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,490 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,490 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,490 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:19,490 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,492 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5157ad9b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,496 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,500 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,500 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,502 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,502 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,502 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,507 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,507 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362519546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 576 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362519546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362519554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362519557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,607 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,610 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,613 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:19,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,620 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3586b1c9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,623 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,627 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,627 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,627 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,634 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,635 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,635 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,735 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,738 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:19,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,745 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664b9a58[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,747 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,747 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,747 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362519748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 578 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362519750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,752 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,752 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362519757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,758 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,759 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,759 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,759 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:19,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362519759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,836 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e5e14cd4cb6743ad8efc2df8f299bf76 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e5e14cd4cb6743ad8efc2df8f299bf76 2023-04-24 18:54:19,859 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,862 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,864 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,866 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,866 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,866 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:19,866 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,869 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5caabf[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:19,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:19,873 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:19,873 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,873 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,874 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:19,876 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:19,876 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:19,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,879 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,885 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:19,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:19,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:19,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:19,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:19,885 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,885 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:19,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,986 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:19,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:19,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:19,989 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:19,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:19,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:19,990 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,990 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,991 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:19,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:19,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:19,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:19,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:19,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:19,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:19,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:19,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:19,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:19,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:19,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:19,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130bfbba[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:19,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,002 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,003 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,003 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,003 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,004 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,004 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,007 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,007 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,009 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,013 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,013 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,013 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,013 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,013 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,013 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362520052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 580 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362520052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362520058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362520062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,113 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,116 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,117 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,117 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,119 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,121 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,121 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,121 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:20,121 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:20,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,123 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,123 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138318eb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:20,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:20,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,129 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,129 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,132 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,136 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,136 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,136 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,236 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,240 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,242 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=655, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/807347a5870c4e66bf90522737c57c55 2023-04-24 18:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2920d999[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:20,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:20,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/f16455e49d9d48828cbda00ef99190e2 2023-04-24 18:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,256 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,256 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,266 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,266 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,366 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,369 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,372 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,373 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,373 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,374 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,374 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,374 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:20,374 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:20,376 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:20,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33338750[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:20,377 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:20,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,379 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,383 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,383 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,387 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,387 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,391 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:20,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,392 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,392 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,492 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,495 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,497 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f297ca7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:20,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:20,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,508 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,508 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,508 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,513 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,513 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,513 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,514 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,514 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:20,514 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362520559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 582 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362520559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362520561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362520567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,614 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,618 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,620 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,622 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,622 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,622 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:20,623 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:20,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:20,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:20,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@746667e9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:20,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:20,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:20,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:20,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:20,631 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:20,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:20,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:20,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:20,631 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:20,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:20,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:20,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,634 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,635 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,637 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:20,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:20,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:20,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,637 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:20,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:20,638 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:20,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/15a4a6b89b4842c88a8440fedc171622 2023-04-24 18:54:20,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/807347a5870c4e66bf90522737c57c55 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55 2023-04-24 18:54:20,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55, entries=400, sequenceid=655, filesize=59.8 K 2023-04-24 18:54:20,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/f16455e49d9d48828cbda00ef99190e2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2 2023-04-24 18:54:20,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2, entries=150, sequenceid=655, filesize=11.9 K 2023-04-24 18:54:20,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/15a4a6b89b4842c88a8440fedc171622 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622 2023-04-24 18:54:20,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622, entries=150, sequenceid=655, filesize=11.9 K 2023-04-24 18:54:20,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1268ms, sequenceid=655, compaction requested=true 2023-04-24 18:54:20,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:20,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:20,688 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:20,688 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:20,688 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:20,688 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:20,689 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:20,689 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:20,689 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,689 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/d68cd23f57c24fa6b777f751a0694485, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.1 K 2023-04-24 18:54:20,689 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113756 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:20,689 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:20,690 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,690 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d68cd23f57c24fa6b777f751a0694485, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:20,690 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b913cb745ba446aac097bcaaf91a250, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=111.1 K 2023-04-24 18:54:20,690 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ad0f0023a5fd4710b40f29ce1433854a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1682362457255 2023-04-24 18:54:20,690 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f16455e49d9d48828cbda00ef99190e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459411 2023-04-24 18:54:20,691 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6b913cb745ba446aac097bcaaf91a250, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:20,691 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 84956f71e616468999e4d0ca3332909f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1682362457255 2023-04-24 18:54:20,691 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 807347a5870c4e66bf90522737c57c55, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459409 2023-04-24 18:54:20,697 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:20,716 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#160 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:20,719 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/6fd67c45718a4f3fa5f1d2ce209947bd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6fd67c45718a4f3fa5f1d2ce209947bd 2023-04-24 18:54:20,724 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into 6fd67c45718a4f3fa5f1d2ce209947bd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:20,724 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:20,724 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362460688; duration=0sec 2023-04-24 18:54:20,724 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:20,724 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:20,725 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:20,726 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:20,726 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,726 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/036367661a3f4f4e9aae48fdaafe77f6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.1 K 2023-04-24 18:54:20,726 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 036367661a3f4f4e9aae48fdaafe77f6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1682362457227 2023-04-24 18:54:20,727 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 058b63bc5a954aab83ff3baafb1c9643, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1682362457255 2023-04-24 18:54:20,727 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 15a4a6b89b4842c88a8440fedc171622, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459411 2023-04-24 18:54:20,738 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:20,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:20,739 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#161 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:20,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:20,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:20,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:20,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:20,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:20,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:20,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:20,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:20,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:20,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:20,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:20,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:20,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:20,760 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:20,761 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:20,769 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/94452acecabb491c94a5b5bfd62be6bf as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/94452acecabb491c94a5b5bfd62be6bf 2023-04-24 18:54:20,776 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 94452acecabb491c94a5b5bfd62be6bf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:20,776 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:20,776 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362460688; duration=0sec 2023-04-24 18:54:20,776 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:21,190 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042461e2e9559f264005aa43c165cb863d64 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042461e2e9559f264005aa43c165cb863d64 2023-04-24 18:54:21,552 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/6821aca4e38c49dda9f0c924f108a19e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6821aca4e38c49dda9f0c924f108a19e 2023-04-24 18:54:21,557 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into 6821aca4e38c49dda9f0c924f108a19e(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:21,557 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:21,557 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362460687; duration=0sec 2023-04-24 18:54:21,557 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:21,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:21,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:21,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362521576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362521577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362521578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 587 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362521578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,596 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=681, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/e9eb41d3c30841e6bc611d8b66c8f877 2023-04-24 18:54:21,613 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/c250d3800f9044348ca9bbd4be32d713 2023-04-24 18:54:21,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362521679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362521680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362521682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 589 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362521682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362521881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362521885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362521885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 591 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362521885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,025 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/8fc2ec9271d04d9ab271cc73d046d6bb 2023-04-24 18:54:22,029 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/e9eb41d3c30841e6bc611d8b66c8f877 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877 2023-04-24 18:54:22,034 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877, entries=150, sequenceid=681, filesize=25.4 K 2023-04-24 18:54:22,035 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/c250d3800f9044348ca9bbd4be32d713 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713 2023-04-24 18:54:22,040 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713, entries=150, sequenceid=681, filesize=11.9 K 2023-04-24 18:54:22,043 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/8fc2ec9271d04d9ab271cc73d046d6bb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb 2023-04-24 18:54:22,047 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb, entries=150, sequenceid=681, filesize=11.9 K 2023-04-24 18:54:22,048 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=60.38 KB/61830 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1288ms, sequenceid=681, compaction requested=false 2023-04-24 18:54:22,049 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:22,049 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,056 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,056 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,058 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba584ed[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,060 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,060 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,066 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,068 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,071 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,071 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,071 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,071 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,071 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,076 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,082 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,082 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,087 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,087 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,187 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:22,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:54:22,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:22,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:22,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:22,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:22,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:22,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:22,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,191 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,193 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,220 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,220 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,222 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,225 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042498903b5a5ccf49f7bb34a199b0aa7cf0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498903b5a5ccf49f7bb34a199b0aa7cf0 2023-04-24 18:54:22,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740391ce[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,227 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362522226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362522227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362522228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 598 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362522229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,233 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,234 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,242 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,242 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362522331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362522331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362522332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 600 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362522332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,342 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,345 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,345 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,345 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,345 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,347 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,352 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,353 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407b17ad[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,356 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,360 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,360 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,372 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,372 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,372 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,472 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,476 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,477 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,479 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,479 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e57964[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,496 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,496 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,499 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,506 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,506 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362522532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362522534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 602 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362522534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362522534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,607 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,610 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,613 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,614 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,615 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,617 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70901799[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,620 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,622 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,625 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,625 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,634 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,634 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,635 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=696, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/5dfbf2dc65194b3a83623ca749d6654a 2023-04-24 18:54:22,734 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,738 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,741 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,741 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,741 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187565a4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,752 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,752 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,754 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,754 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,759 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,760 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,760 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362522835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362522837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362522837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:22,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 604 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362522839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,860 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,863 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,867 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:22,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:22,868 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:22,870 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abcf78[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:22,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:22,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:22,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:22,874 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,874 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:22,874 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,874 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:22,876 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:22,877 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:22,877 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:22,877 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:22,877 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:22,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:22,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:22,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:22,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,884 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:22,885 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:22,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:22,885 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:22,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:22,985 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:22,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:22,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:22,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:22,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:22,988 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:22,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:22,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:22,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:22,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:22,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,990 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,990 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:22,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:22,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:22,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:22,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:23,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:23,011 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,011 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:23,011 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:23,011 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:23,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,019 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca9a850[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:23,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:23,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:23,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:23,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:23,024 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:23,027 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:23,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:23,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:23,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:23,027 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:23,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,030 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,032 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:23,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:23,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,036 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,036 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:23,036 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/94dd4947f2364bf7bf9f04e45056208b 2023-04-24 18:54:23,136 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:23,140 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:23,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:23,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:23,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,142 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:23,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:23,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:23,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf54e83[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:23,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:23,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,149 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:23,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:23,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:23,154 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:23,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:23,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:23,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:23,154 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:23,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:23,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,161 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:23,161 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:23,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,261 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:23,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:23,264 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:23,264 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,264 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:23,264 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:23,265 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,265 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:23,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:23,267 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:23,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:23,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:23,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:23,270 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f9e48fc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:23,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:23,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,273 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:23,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:23,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:23,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:23,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:23,277 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:23,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:23,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:23,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:23,277 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:23,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,279 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,280 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:23,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:23,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:23,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,284 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,284 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:23,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362523339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:23,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:23,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362523341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362523341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:23,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 606 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362523345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,384 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:23,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:23,387 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:23,387 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:23,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:23,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,389 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:23,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:23,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:23,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:23,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:23,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:23,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:23,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31908a78[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:23,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:23,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:23,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:23,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:23,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:23,407 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:23,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:23,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:23,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:23,407 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:23,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:23,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:23,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:23,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:23,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,413 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:23,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:23,413 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:23,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:23,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:23,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/78c2a58c0f0f4a91bb65eb196f65b11f 2023-04-24 18:54:23,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/5dfbf2dc65194b3a83623ca749d6654a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a 2023-04-24 18:54:23,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a, entries=150, sequenceid=696, filesize=25.4 K 2023-04-24 18:54:23,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/94dd4947f2364bf7bf9f04e45056208b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b 2023-04-24 18:54:23,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b, entries=100, sequenceid=696, filesize=9.5 K 2023-04-24 18:54:23,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/78c2a58c0f0f4a91bb65eb196f65b11f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f 2023-04-24 18:54:23,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f, entries=100, sequenceid=696, filesize=9.5 K 2023-04-24 18:54:23,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1296ms, sequenceid=696, compaction requested=true 2023-04-24 18:54:23,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:23,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:23,484 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:23,484 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:23,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:23,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:23,486 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:23,486 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:23,486 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:23,486 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,486 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6821aca4e38c49dda9f0c924f108a19e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=76.8 K 2023-04-24 18:54:23,486 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:23,486 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,486 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6fd67c45718a4f3fa5f1d2ce209947bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=33.8 K 2023-04-24 18:54:23,487 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6821aca4e38c49dda9f0c924f108a19e, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459411 2023-04-24 18:54:23,487 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6fd67c45718a4f3fa5f1d2ce209947bd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459411 2023-04-24 18:54:23,487 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e9eb41d3c30841e6bc611d8b66c8f877, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682362459446 2023-04-24 18:54:23,488 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c250d3800f9044348ca9bbd4be32d713, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682362459446 2023-04-24 18:54:23,488 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5dfbf2dc65194b3a83623ca749d6654a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362461573 2023-04-24 18:54:23,488 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 94dd4947f2364bf7bf9f04e45056208b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362461577 2023-04-24 18:54:23,497 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:23,501 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#168 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:23,513 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:23,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:23,516 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:23,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:23,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:23,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,519 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:23,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:23,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:23,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:23,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:23,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:23,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:23,520 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:23,521 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:23,908 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/25ff0b232e2b4657a30c0cb5575c4479 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/25ff0b232e2b4657a30c0cb5575c4479 2023-04-24 18:54:23,914 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into 25ff0b232e2b4657a30c0cb5575c4479(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:23,914 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:23,915 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362463484; duration=0sec 2023-04-24 18:54:23,915 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:23,915 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:23,917 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:23,917 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:23,917 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:23,917 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/94452acecabb491c94a5b5bfd62be6bf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=33.8 K 2023-04-24 18:54:23,917 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 94452acecabb491c94a5b5bfd62be6bf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682362459411 2023-04-24 18:54:23,918 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8fc2ec9271d04d9ab271cc73d046d6bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1682362459446 2023-04-24 18:54:23,918 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 78c2a58c0f0f4a91bb65eb196f65b11f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362461577 2023-04-24 18:54:23,926 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:23,935 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/8700f0e5562845db9787b92302cd14a4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8700f0e5562845db9787b92302cd14a4 2023-04-24 18:54:23,937 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424f622cf6d74434d92986c4d78e9e61fb1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f622cf6d74434d92986c4d78e9e61fb1 2023-04-24 18:54:23,948 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 8700f0e5562845db9787b92302cd14a4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:23,948 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:23,948 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362463484; duration=0sec 2023-04-24 18:54:23,948 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:23,955 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=715, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/c4ba6fcfc80d416195078ff26ce3065e 2023-04-24 18:54:24,331 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/12b0d49dc8004c9ba963d2b91eabfcd9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/12b0d49dc8004c9ba963d2b91eabfcd9 2023-04-24 18:54:24,336 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into 12b0d49dc8004c9ba963d2b91eabfcd9(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:24,336 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:24,336 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362463484; duration=0sec 2023-04-24 18:54:24,336 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:24,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:24,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:24,367 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/6e92e809844b448fba590e8d15d16a04 2023-04-24 18:54:24,385 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/a77e2dbdf7754b9393fbd54364bb60d5 2023-04-24 18:54:24,389 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/c4ba6fcfc80d416195078ff26ce3065e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e 2023-04-24 18:54:24,393 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e, entries=150, sequenceid=715, filesize=25.4 K 2023-04-24 18:54:24,393 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/6e92e809844b448fba590e8d15d16a04 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04 2023-04-24 18:54:24,397 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04, entries=150, sequenceid=715, filesize=11.9 K 2023-04-24 18:54:24,398 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/a77e2dbdf7754b9393fbd54364bb60d5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5 2023-04-24 18:54:24,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362524359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,404 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5, entries=150, sequenceid=715, filesize=11.9 K 2023-04-24 18:54:24,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362524400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,406 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 103750326d6eb6a8536f7e1c9d4b7a6f in 886ms, sequenceid=715, compaction requested=false 2023-04-24 18:54:24,406 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:24,406 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:24,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:24,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:24,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:24,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@509a4f32[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:24,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:24,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:24,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:24,415 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:24,420 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:24,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:24,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:24,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:24,421 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:24,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362524428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 618 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362524429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:24,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:24,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:24,432 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:24,432 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:24,445 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ad4d6ec4ca364a3683e1d58eeb3e0d4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ad4d6ec4ca364a3683e1d58eeb3e0d4a 2023-04-24 18:54:24,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362524501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362524505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362524530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 620 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362524530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,532 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:24,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:24,535 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:24,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:24,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:24,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,537 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:24,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:24,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:24,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:24,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:24,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:24,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:24,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:24,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266d82f4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:24,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,545 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:24,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:24,547 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:24,549 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:24,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:24,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:24,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:24,550 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:24,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,555 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:24,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:24,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:24,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,556 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:24,556 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,656 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:24,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:24,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:24,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:24,659 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:24,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:24,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:24,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:24,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:24,663 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,663 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:24,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:24,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:24,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba9e76d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:24,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:24,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:24,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:24,674 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:24,677 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:24,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:24,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:24,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:24,677 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:24,678 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,678 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,678 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:24,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,683 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:24,683 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:24,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362524704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362524708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362524733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:24,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 622 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362524734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,786 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:24,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:24,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:24,790 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:24,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:24,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:24,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,791 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:24,796 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,797 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@edf792a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:24,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:24,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:24,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:24,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:24,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:24,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:24,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:24,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:24,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:24,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:24,805 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,811 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,811 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,811 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:24,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,812 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:24,812 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:24,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,854 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=735, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/4170c2615e50496084461a7a6877befa 2023-04-24 18:54:24,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/b546e0cfddf245e0b0626b39bae90082 2023-04-24 18:54:24,912 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:24,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:24,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:24,916 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:24,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:24,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:24,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,918 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:24,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:24,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:24,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:24,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,921 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:24,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3e441c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:24,923 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:24,923 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,924 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,924 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:24,924 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:24,924 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,924 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:24,924 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:24,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:24,928 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:24,928 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:24,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:24,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:24,929 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:24,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:24,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,932 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:24,934 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:24,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:24,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:24,934 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:24,934 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:24,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362525008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362525013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,034 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:25,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:25,038 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:25,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:25,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:25,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362525038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 624 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362525038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:25,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:25,045 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468e1d4d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:25,050 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:25,050 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,051 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:25,051 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:25,051 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,051 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:25,052 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:25,052 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,052 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:25,055 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:25,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:25,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:25,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:25,055 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:25,057 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,057 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,057 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,057 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,059 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,065 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:25,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,065 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:25,065 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:25,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,166 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:25,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:25,169 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:25,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:25,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:25,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:25,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:25,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:25,173 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,173 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:25,173 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:25,173 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:25,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,175 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e8187e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:25,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:25,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:25,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:25,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:25,179 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:25,180 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,180 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:25,182 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:25,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:25,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:25,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:25,182 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:25,184 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,184 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,184 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,185 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,186 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,189 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:25,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:25,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,190 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:25,190 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d3e26246c69a4080988046916285c0cc 2023-04-24 18:54:25,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/4170c2615e50496084461a7a6877befa as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa 2023-04-24 18:54:25,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa, entries=150, sequenceid=735, filesize=25.4 K 2023-04-24 18:54:25,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/b546e0cfddf245e0b0626b39bae90082 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082 2023-04-24 18:54:25,290 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:25,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:25,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082, entries=150, sequenceid=735, filesize=11.9 K 2023-04-24 18:54:25,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/d3e26246c69a4080988046916285c0cc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc 2023-04-24 18:54:25,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:25,293 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:25,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:25,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:25,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,295 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc, entries=150, sequenceid=735, filesize=11.9 K 2023-04-24 18:54:25,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,296 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 103750326d6eb6a8536f7e1c9d4b7a6f in 888ms, sequenceid=735, compaction requested=true 2023-04-24 18:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:25,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:25,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:25,296 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:25,296 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:25,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:25,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:25,297 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78752 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:25,297 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:25,298 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:25,298 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,298 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:25,298 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/12b0d49dc8004c9ba963d2b91eabfcd9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=76.9 K 2023-04-24 18:54:25,298 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,298 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/25ff0b232e2b4657a30c0cb5575c4479, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.3 K 2023-04-24 18:54:25,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:25,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:25,298 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,298 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:25,299 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:25,299 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 12b0d49dc8004c9ba963d2b91eabfcd9, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362459452 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:25,299 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:25,299 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 25ff0b232e2b4657a30c0cb5575c4479, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362459452 2023-04-24 18:54:25,299 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c4ba6fcfc80d416195078ff26ce3065e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1682362462224 2023-04-24 18:54:25,300 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6e92e809844b448fba590e8d15d16a04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1682362462224 2023-04-24 18:54:25,300 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4170c2615e50496084461a7a6877befa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:25,300 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b546e0cfddf245e0b0626b39bae90082, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:25,309 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#179 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:25,328 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#178 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:25,331 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424168e8739e23947c2b679c93fc915046c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424168e8739e23947c2b679c93fc915046c 2023-04-24 18:54:25,337 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/e6f1a9b221554b2790e23712d6e7c6ea as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e6f1a9b221554b2790e23712d6e7c6ea 2023-04-24 18:54:25,343 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into e6f1a9b221554b2790e23712d6e7c6ea(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:25,343 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:25,343 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362465296; duration=0sec 2023-04-24 18:54:25,343 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:25,343 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:25,343 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=750, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/6b8f04fb484441a5a7ff4433e8ef208c 2023-04-24 18:54:25,344 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:25,345 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:25,345 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,345 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8700f0e5562845db9787b92302cd14a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.3 K 2023-04-24 18:54:25,345 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8700f0e5562845db9787b92302cd14a4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1682362459452 2023-04-24 18:54:25,346 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a77e2dbdf7754b9393fbd54364bb60d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=715, earliestPutTs=1682362462224 2023-04-24 18:54:25,346 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d3e26246c69a4080988046916285c0cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:25,358 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#181 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:25,376 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/e2995374a4b742279d2ea2e9507d8fdf as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/e2995374a4b742279d2ea2e9507d8fdf 2023-04-24 18:54:25,382 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into e2995374a4b742279d2ea2e9507d8fdf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:25,382 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:25,382 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362465297; duration=0sec 2023-04-24 18:54:25,382 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:25,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:25,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:25,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 626 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362525544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362525544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362525547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362525548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362525648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362525650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362525650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,772 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/9c0bddfb41d4438b951c1d524ad1d15e 2023-04-24 18:54:25,801 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/51da7196b60a48f1a356ac96f3f34126 2023-04-24 18:54:25,806 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/6b8f04fb484441a5a7ff4433e8ef208c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c 2023-04-24 18:54:25,810 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c, entries=150, sequenceid=750, filesize=25.4 K 2023-04-24 18:54:25,811 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/9c0bddfb41d4438b951c1d524ad1d15e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e 2023-04-24 18:54:25,817 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e, entries=150, sequenceid=750, filesize=11.9 K 2023-04-24 18:54:25,818 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/51da7196b60a48f1a356ac96f3f34126 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126 2023-04-24 18:54:25,822 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126, entries=150, sequenceid=750, filesize=11.9 K 2023-04-24 18:54:25,823 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 103750326d6eb6a8536f7e1c9d4b7a6f in 525ms, sequenceid=750, compaction requested=false 2023-04-24 18:54:25,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:25,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:25,827 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:25,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470b2120[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:25,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:25,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:25,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:25,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,831 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:25,833 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:25,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:25,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:25,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:25,833 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:25,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:25,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,841 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,841 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:25,841 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:25,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:25,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:25,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:25,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362525885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362525886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362525890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,898 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424c90d1bb3d7994005bde5fe34add24f9f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c90d1bb3d7994005bde5fe34add24f9f 2023-04-24 18:54:25,916 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=773, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/0e023b6da39c4f46bb9920498d7d4335 2023-04-24 18:54:25,941 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:25,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:25,944 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:25,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:25,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,947 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:25,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:25,954 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:25,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39eb8379[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:25,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:25,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:25,957 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:25,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:25,962 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:25,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:25,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:25,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:25,962 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:25,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:25,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:25,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,966 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:25,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:25,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:25,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:25,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:25,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:25,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:25,971 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:25,971 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:25,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:25,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362525991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362525992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:25,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362525995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,071 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,074 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,081 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,081 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,081 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f39957c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,084 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,086 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,088 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,088 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,092 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,093 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,094 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,096 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,097 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:26,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,097 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,097 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362526194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,197 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362526195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362526200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,201 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,201 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,201 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,201 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,206 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,206 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,207 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,207 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,207 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,207 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,209 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,211 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d60470d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,211 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,215 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,219 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,219 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,230 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,231 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,331 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/4fcfe260fd8e480d90bf82d2459807c2 2023-04-24 18:54:26,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,335 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,335 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,335 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,335 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,342 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,342 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,343 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,343 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63fc7ba0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,347 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,349 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,351 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,351 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,354 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,354 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,354 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:26,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,360 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,360 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,460 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,464 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,464 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,464 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,464 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,464 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,464 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,469 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,470 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,470 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,470 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,470 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669cced[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,474 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,476 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,477 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,480 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,480 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,483 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,488 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,488 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,488 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43698 deadline: 1682362526499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43680 deadline: 1682362526501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43676 deadline: 1682362526501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,550 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/b4f949a749c944a88b47668a0da7d9a4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/b4f949a749c944a88b47668a0da7d9a4 2023-04-24 18:54:26,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 628 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362526549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,555 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into b4f949a749c944a88b47668a0da7d9a4(size=26.1 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 18:54:26,555 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:26,555 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362465296; duration=1sec 2023-04-24 18:54:26,555 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:26,589 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,591 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,592 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,592 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,592 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,592 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,592 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,594 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,595 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,595 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,595 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,595 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,597 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b82b29[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,600 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,602 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,602 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,602 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,602 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,604 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,605 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,610 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,610 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:26,711 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,714 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,716 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,716 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,718 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,718 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,718 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:26,718 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:26,718 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:26,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:26,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:26,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:26,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1f54bc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:26,728 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:26,728 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,729 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:26,730 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,730 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:26,732 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:26,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:26,734 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:26,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:26,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:26,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:26,734 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:26,736 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,736 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:26,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,736 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:26,736 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:26,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:26,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:26,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:26,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,742 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:26,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,742 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:26,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:26,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/248ed6e5a7eb44279f3b546890829aff 2023-04-24 18:54:26,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/0e023b6da39c4f46bb9920498d7d4335 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335 2023-04-24 18:54:26,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335, entries=150, sequenceid=773, filesize=25.4 K 2023-04-24 18:54:26,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/4fcfe260fd8e480d90bf82d2459807c2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2 2023-04-24 18:54:26,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2, entries=150, sequenceid=773, filesize=11.9 K 2023-04-24 18:54:26,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/248ed6e5a7eb44279f3b546890829aff as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff 2023-04-24 18:54:26,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff, entries=150, sequenceid=773, filesize=11.9 K 2023-04-24 18:54:26,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 103750326d6eb6a8536f7e1c9d4b7a6f in 892ms, sequenceid=773, compaction requested=true 2023-04-24 18:54:26,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:26,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:26,768 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:26,768 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:26,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:26,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:26,770 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:26,770 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:26,770 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:26,770 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:26,770 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,771 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,771 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/b4f949a749c944a88b47668a0da7d9a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=77.0 K 2023-04-24 18:54:26,771 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e6f1a9b221554b2790e23712d6e7c6ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.4 K 2023-04-24 18:54:26,771 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b4f949a749c944a88b47668a0da7d9a4, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:26,771 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6b8f04fb484441a5a7ff4433e8ef208c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1682362464422 2023-04-24 18:54:26,772 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0e023b6da39c4f46bb9920498d7d4335, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:26,772 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e6f1a9b221554b2790e23712d6e7c6ea, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:26,772 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9c0bddfb41d4438b951c1d524ad1d15e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1682362464422 2023-04-24 18:54:26,773 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4fcfe260fd8e480d90bf82d2459807c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:26,780 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:26,787 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#186 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:26,805 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/a00109df74fc46029cb81688875a1d65 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a00109df74fc46029cb81688875a1d65 2023-04-24 18:54:26,810 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into a00109df74fc46029cb81688875a1d65(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:26,810 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:26,810 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=13, startTime=1682362466768; duration=0sec 2023-04-24 18:54:26,810 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:26,810 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:26,811 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:26,811 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:26,811 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,812 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/e2995374a4b742279d2ea2e9507d8fdf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=36.4 K 2023-04-24 18:54:26,812 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e2995374a4b742279d2ea2e9507d8fdf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1682362464363 2023-04-24 18:54:26,812 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 51da7196b60a48f1a356ac96f3f34126, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1682362464422 2023-04-24 18:54:26,813 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 248ed6e5a7eb44279f3b546890829aff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:26,819 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:26,832 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/a2aec996128d48acbd25f45a27cdadd1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a2aec996128d48acbd25f45a27cdadd1 2023-04-24 18:54:26,837 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into a2aec996128d48acbd25f45a27cdadd1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:26,837 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:26,837 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=13, startTime=1682362466768; duration=0sec 2023-04-24 18:54:26,837 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:26,842 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:26,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:26,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:26,846 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:26,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:26,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:26,847 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,847 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,848 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:26,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:26,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:26,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:26,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:26,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:26,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:26,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:26,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:26,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:26,856 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:26,856 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:26,856 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:54:26,856 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:26,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:26,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:26,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:26,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:26,857 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:27,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:27,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362467005, max timestamp=1682362467038Num uniques 859; 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362467005, max timestamp=1682362467038Num uniques 859; 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362467005, max timestamp=1682362467038Num uniques 859; 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:27,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:27,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362527056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362527057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362527058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362527163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362527163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362527167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,281 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248f8510126cdf4318a41cd269ae20d4ce to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f8510126cdf4318a41cd269ae20d4ce 2023-04-24 18:54:27,288 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=787, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a14fd3aa2b6f469a8bbb00b76341aa95 2023-04-24 18:54:27,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43668 deadline: 1682362527335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,341 DEBUG [Thread-850] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18264 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:27,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362527366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362527366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362527369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,609 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/c90668c3b485453eb5a1979ae77aca5a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c90668c3b485453eb5a1979ae77aca5a 2023-04-24 18:54:27,615 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into c90668c3b485453eb5a1979ae77aca5a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:27,615 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:27,615 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=13, startTime=1682362466768; duration=0sec 2023-04-24 18:54:27,615 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:27,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362527669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362527670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:27,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362527672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,702 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/e50efba9323343b4b73c0762b9cb418f 2023-04-24 18:54:27,713 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/fa70af7965a0493b80ec83d943a1ad07 2023-04-24 18:54:27,717 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/a14fd3aa2b6f469a8bbb00b76341aa95 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95 2023-04-24 18:54:27,721 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95, entries=150, sequenceid=787, filesize=25.4 K 2023-04-24 18:54:27,722 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/e50efba9323343b4b73c0762b9cb418f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f 2023-04-24 18:54:27,725 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f, entries=150, sequenceid=787, filesize=11.9 K 2023-04-24 18:54:27,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/fa70af7965a0493b80ec83d943a1ad07 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07 2023-04-24 18:54:27,732 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07, entries=150, sequenceid=787, filesize=11.9 K 2023-04-24 18:54:27,733 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 103750326d6eb6a8536f7e1c9d4b7a6f in 877ms, sequenceid=787, compaction requested=false 2023-04-24 18:54:27,733 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:27,733 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:27,735 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:27,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:27,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:27,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3abe1924[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:27,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:27,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:27,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:27,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:27,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:27,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:27,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:27,746 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:27,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:27,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:27,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:27,746 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:27,747 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:27,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:27,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:27,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:27,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:27,753 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:27,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:27,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,753 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:27,754 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:27,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:27,854 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:27,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:27,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:27,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:27,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:27,857 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:27,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:27,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:27,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:27,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:27,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,860 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:27,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:27,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:27,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:27,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:27,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:27,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:27,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:27,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. started... 2023-04-24 18:54:27,862 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:27,862 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:27,879 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424cf712844101c4d028030f58aa60763ef to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cf712844101c4d028030f58aa60763ef 2023-04-24 18:54:28,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:28,175 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. as already flushing 2023-04-24 18:54:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362528181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362528182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362528185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,288 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=822, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/7c4749381c0b46b9ab976c6816eb23e8 2023-04-24 18:54:28,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362528286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362528287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362528289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,301 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=822 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/55040097d108472aa2af15796ab305a2 2023-04-24 18:54:28,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43676 deadline: 1682362528490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43680 deadline: 1682362528491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43698 deadline: 1682362528491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 630 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43638 deadline: 1682362528553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,557 DEBUG [Thread-858] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:28,718 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=822 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/a94b4d816ad2415eb735696849bb74b4 2023-04-24 18:54:28,722 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/7c4749381c0b46b9ab976c6816eb23e8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8 2023-04-24 18:54:28,726 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8, entries=150, sequenceid=822, filesize=25.4 K 2023-04-24 18:54:28,727 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/55040097d108472aa2af15796ab305a2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2 2023-04-24 18:54:28,731 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2, entries=150, sequenceid=822, filesize=11.9 K 2023-04-24 18:54:28,731 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/a94b4d816ad2415eb735696849bb74b4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4 2023-04-24 18:54:28,737 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4, entries=150, sequenceid=822, filesize=11.9 K 2023-04-24 18:54:28,738 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 103750326d6eb6a8536f7e1c9d4b7a6f in 877ms, sequenceid=822, compaction requested=true 2023-04-24 18:54:28,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:28,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:28,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f8b43a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:28,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:28,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:28,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:28,745 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d325f4 to 127.0.0.1:59583 2023-04-24 18:54:28,745 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,745 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:28,745 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:59583 2023-04-24 18:54:28,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:28,746 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:28,746 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:59583 2023-04-24 18:54:28,746 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:28,747 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:59583 2023-04-24 18:54:28,747 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:28,750 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:28,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:28,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:28,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:28,750 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:28,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:28,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,753 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:28,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:28,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:28,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:28,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:28,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:28,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:28,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:28,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:28,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:28,757 DEBUG [Thread-860] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:28,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:28,757 DEBUG [Thread-860] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:28,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:28,794 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:54:28,794 DEBUG [Thread-854] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:59583 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:28,794 DEBUG [Thread-854] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:28,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:28,795 DEBUG [Thread-856] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:59583 2023-04-24 18:54:28,795 DEBUG [Thread-856] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,798 DEBUG [Thread-852] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:59583 2023-04-24 18:54:28,798 DEBUG [Thread-852] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:28,857 DEBUG [Thread-860] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:28,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:29,209 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424c23f15fb17524e2daaea3545384ddf60 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c23f15fb17524e2daaea3545384ddf60 2023-04-24 18:54:29,614 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=834, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/8c1ba863fa9c4e4582ba90d9ba08b61e 2023-04-24 18:54:30,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/bbb6f3e5fdae4661be62a7b590c99aef 2023-04-24 18:54:30,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/9c7c492860c04f62abf7e17aafb5560b 2023-04-24 18:54:30,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/8c1ba863fa9c4e4582ba90d9ba08b61e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e 2023-04-24 18:54:30,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e, entries=150, sequenceid=834, filesize=25.4 K 2023-04-24 18:54:30,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/bbb6f3e5fdae4661be62a7b590c99aef as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef 2023-04-24 18:54:30,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef, entries=150, sequenceid=834, filesize=11.9 K 2023-04-24 18:54:30,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/9c7c492860c04f62abf7e17aafb5560b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b 2023-04-24 18:54:30,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b, entries=150, sequenceid=834, filesize=11.9 K 2023-04-24 18:54:30,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1655ms, sequenceid=834, compaction requested=true 2023-04-24 18:54:30,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:30,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:30,449 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:30,449 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:30,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:30,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:30,450 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:30,451 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/A is initiating minor compaction (all files) 2023-04-24 18:54:30,451 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:30,451 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/A in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:30,451 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/B is initiating minor compaction (all files) 2023-04-24 18:54:30,451 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/B in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:30,451 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c90668c3b485453eb5a1979ae77aca5a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=102.5 K 2023-04-24 18:54:30,451 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a00109df74fc46029cb81688875a1d65, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=48.4 K 2023-04-24 18:54:30,451 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c90668c3b485453eb5a1979ae77aca5a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:30,451 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a00109df74fc46029cb81688875a1d65, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:30,452 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a14fd3aa2b6f469a8bbb00b76341aa95, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682362465882 2023-04-24 18:54:30,452 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e50efba9323343b4b73c0762b9cb418f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682362465882 2023-04-24 18:54:30,452 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 55040097d108472aa2af15796ab305a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1682362467053 2023-04-24 18:54:30,452 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7c4749381c0b46b9ab976c6816eb23e8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1682362467053 2023-04-24 18:54:30,453 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting bbb6f3e5fdae4661be62a7b590c99aef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682362468181 2023-04-24 18:54:30,453 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8c1ba863fa9c4e4582ba90d9ba08b61e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682362468181 2023-04-24 18:54:30,460 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#B#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:30,467 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#A#compaction#199 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:30,469 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/a8afc7c3b80449159fe628b68ba73a48 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a8afc7c3b80449159fe628b68ba73a48 2023-04-24 18:54:30,475 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/B of 103750326d6eb6a8536f7e1c9d4b7a6f into a8afc7c3b80449159fe628b68ba73a48(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:30,475 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:30,475 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/B, priority=12, startTime=1682362470449; duration=0sec 2023-04-24 18:54:30,475 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:30,476 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:30,477 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:30,477 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 103750326d6eb6a8536f7e1c9d4b7a6f/C is initiating minor compaction (all files) 2023-04-24 18:54:30,477 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 103750326d6eb6a8536f7e1c9d4b7a6f/C in TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:30,477 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a2aec996128d48acbd25f45a27cdadd1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp, totalSize=48.4 K 2023-04-24 18:54:30,477 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a2aec996128d48acbd25f45a27cdadd1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1682362465852 2023-04-24 18:54:30,478 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fa70af7965a0493b80ec83d943a1ad07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682362465882 2023-04-24 18:54:30,478 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a94b4d816ad2415eb735696849bb74b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=822, earliestPutTs=1682362467053 2023-04-24 18:54:30,478 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9c7c492860c04f62abf7e17aafb5560b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682362468181 2023-04-24 18:54:30,485 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 103750326d6eb6a8536f7e1c9d4b7a6f#C#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:30,893 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/46407996925e47dcbeda9073dad24c33 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/46407996925e47dcbeda9073dad24c33 2023-04-24 18:54:30,896 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/C of 103750326d6eb6a8536f7e1c9d4b7a6f into 46407996925e47dcbeda9073dad24c33(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:30,897 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:30,897 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/C, priority=12, startTime=1682362470449; duration=0sec 2023-04-24 18:54:30,897 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:31,684 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/bbea757b52604a5ea119f916f1fdf983 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/bbea757b52604a5ea119f916f1fdf983 2023-04-24 18:54:31,688 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 103750326d6eb6a8536f7e1c9d4b7a6f/A of 103750326d6eb6a8536f7e1c9d4b7a6f into bbea757b52604a5ea119f916f1fdf983(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 18:54:31,688 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:31,688 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f., storeName=103750326d6eb6a8536f7e1c9d4b7a6f/A, priority=12, startTime=1682362470449; duration=1sec 2023-04-24 18:54:31,688 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:32,562 DEBUG [Thread-858] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:59583 2023-04-24 18:54:32,562 DEBUG [Thread-858] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:37,399 DEBUG [Thread-850] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:59583 2023-04-24 18:54:37,399 DEBUG [Thread-850] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:37,399 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:54:37,399 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 533 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5330 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5251 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 2211 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 6633 rows 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 2212 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 6636 rows 2023-04-24 18:54:37,400 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:54:37,400 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x324dc814 to 127.0.0.1:59583 2023-04-24 18:54:37,400 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:54:37,401 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:54:37,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:54:37,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:37,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:37,407 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362477406"}]},"ts":"1682362477406"} 2023-04-24 18:54:37,408 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:54:37,410 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:54:37,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, UNASSIGN}] 2023-04-24 18:54:37,412 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, UNASSIGN 2023-04-24 18:54:37,413 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:37,413 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362477413"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362477413"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362477413"}]},"ts":"1682362477413"} 2023-04-24 18:54:37,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:37,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:37,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 103750326d6eb6a8536f7e1c9d4b7a6f, disabling compactions & flushes 2023-04-24 18:54:37,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. after waiting 0 ms 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:37,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 103750326d6eb6a8536f7e1c9d4b7a6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=A 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=B 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 103750326d6eb6a8536f7e1c9d4b7a6f, store=C 2023-04-24 18:54:37,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:37,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:37,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d3417a57863141f6abf1f97d5de37137 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d3417a57863141f6abf1f97d5de37137 2023-04-24 18:54:38,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:38,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=844, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/2509060d142349f49a23ebf706d99272 2023-04-24 18:54:38,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:38,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/40b61dbe0e7744c1ba879bf5b05f2a28 2023-04-24 18:54:39,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/1da20bd8295b40b7ada2149e9ed8ab63 2023-04-24 18:54:39,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/A/2509060d142349f49a23ebf706d99272 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/2509060d142349f49a23ebf706d99272 2023-04-24 18:54:39,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/2509060d142349f49a23ebf706d99272, entries=150, sequenceid=844, filesize=25.4 K 2023-04-24 18:54:39,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/B/40b61dbe0e7744c1ba879bf5b05f2a28 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/40b61dbe0e7744c1ba879bf5b05f2a28 2023-04-24 18:54:39,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/40b61dbe0e7744c1ba879bf5b05f2a28, entries=150, sequenceid=844, filesize=11.9 K 2023-04-24 18:54:39,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/.tmp/C/1da20bd8295b40b7ada2149e9ed8ab63 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/1da20bd8295b40b7ada2149e9ed8ab63 2023-04-24 18:54:39,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/1da20bd8295b40b7ada2149e9ed8ab63, entries=150, sequenceid=844, filesize=11.9 K 2023-04-24 18:54:39,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 103750326d6eb6a8536f7e1c9d4b7a6f in 1658ms, sequenceid=844, compaction requested=false 2023-04-24 18:54:39,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/3f36f4e911f34eeda749495c5906c746, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a8079503481f4f6d849d9b902063b8f3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b913cb745ba446aac097bcaaf91a250, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6821aca4e38c49dda9f0c924f108a19e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/12b0d49dc8004c9ba963d2b91eabfcd9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/b4f949a749c944a88b47668a0da7d9a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c90668c3b485453eb5a1979ae77aca5a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e] to archive 2023-04-24 18:54:39,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:39,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/999c7ca63c20449f84f5ffd3e2f5d58a 2023-04-24 18:54:39,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/dbb9616973a84f328f129a2cb359f1a7 2023-04-24 18:54:39,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/f110647278dc40fcbcb4cf0d414a45dd 2023-04-24 18:54:39,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a2632c43c40145049882599d98d1673f 2023-04-24 18:54:39,251 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/367c80d31ff24caa90cf617e55d917e8 2023-04-24 18:54:39,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/10b462566a414b0198a865fbdd7b137a 2023-04-24 18:54:39,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/3f36f4e911f34eeda749495c5906c746 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/3f36f4e911f34eeda749495c5906c746 2023-04-24 18:54:39,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/efca336650b4406db10c1ec3a26983ba 2023-04-24 18:54:39,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a8079503481f4f6d849d9b902063b8f3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a8079503481f4f6d849d9b902063b8f3 2023-04-24 18:54:39,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c949189839a14255897460cfd017e6b2 2023-04-24 18:54:39,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5b298951cb474ee5b628d38410e8cfb4 2023-04-24 18:54:39,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a1be131db8c14474ad1422e411e195a9 2023-04-24 18:54:39,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b913cb745ba446aac097bcaaf91a250 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b913cb745ba446aac097bcaaf91a250 2023-04-24 18:54:39,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/84956f71e616468999e4d0ca3332909f 2023-04-24 18:54:39,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/807347a5870c4e66bf90522737c57c55 2023-04-24 18:54:39,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6821aca4e38c49dda9f0c924f108a19e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6821aca4e38c49dda9f0c924f108a19e 2023-04-24 18:54:39,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/e9eb41d3c30841e6bc611d8b66c8f877 2023-04-24 18:54:39,265 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/12b0d49dc8004c9ba963d2b91eabfcd9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/12b0d49dc8004c9ba963d2b91eabfcd9 2023-04-24 18:54:39,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/5dfbf2dc65194b3a83623ca749d6654a 2023-04-24 18:54:39,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c4ba6fcfc80d416195078ff26ce3065e 2023-04-24 18:54:39,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/b4f949a749c944a88b47668a0da7d9a4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/b4f949a749c944a88b47668a0da7d9a4 2023-04-24 18:54:39,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/4170c2615e50496084461a7a6877befa 2023-04-24 18:54:39,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/6b8f04fb484441a5a7ff4433e8ef208c 2023-04-24 18:54:39,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c90668c3b485453eb5a1979ae77aca5a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/c90668c3b485453eb5a1979ae77aca5a 2023-04-24 18:54:39,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/0e023b6da39c4f46bb9920498d7d4335 2023-04-24 18:54:39,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/a14fd3aa2b6f469a8bbb00b76341aa95 2023-04-24 18:54:39,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/7c4749381c0b46b9ab976c6816eb23e8 2023-04-24 18:54:39,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/8c1ba863fa9c4e4582ba90d9ba08b61e 2023-04-24 18:54:39,278 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/fe933f3d0da34c6193df8301d8d400f9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/73f018eb946e4764acdb355c127c2c95, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/d68cd23f57c24fa6b777f751a0694485, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6fd67c45718a4f3fa5f1d2ce209947bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/25ff0b232e2b4657a30c0cb5575c4479, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e6f1a9b221554b2790e23712d6e7c6ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a00109df74fc46029cb81688875a1d65, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef] to archive 2023-04-24 18:54:39,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:39,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/765734fec0f3484996813d98fa9b9f6b 2023-04-24 18:54:39,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ba03a62970564885adce57dd2bae8aef 2023-04-24 18:54:39,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/57ddb3cf4f8148c6a97fd5f7599a855a 2023-04-24 18:54:39,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/7dede12f8d4a429280f0e9d10c5ab879 2023-04-24 18:54:39,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4da6256a54264f37907c07497d8fbeeb 2023-04-24 18:54:39,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/fe933f3d0da34c6193df8301d8d400f9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/fe933f3d0da34c6193df8301d8d400f9 2023-04-24 18:54:39,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/5c898b0a89d346d6bf29f5a91fa9acad 2023-04-24 18:54:39,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/881f0de6e42f49769acd0db1d7da598d 2023-04-24 18:54:39,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/73f018eb946e4764acdb355c127c2c95 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/73f018eb946e4764acdb355c127c2c95 2023-04-24 18:54:39,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/51f89a7dc3d64f5c8a6f318b462d1909 2023-04-24 18:54:39,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ac0f99035e944164943a7b4c50c9ad91 2023-04-24 18:54:39,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/d68cd23f57c24fa6b777f751a0694485 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/d68cd23f57c24fa6b777f751a0694485 2023-04-24 18:54:39,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f6ed334068b44d2895350a1c6c10d274 2023-04-24 18:54:39,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/ad0f0023a5fd4710b40f29ce1433854a 2023-04-24 18:54:39,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6fd67c45718a4f3fa5f1d2ce209947bd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6fd67c45718a4f3fa5f1d2ce209947bd 2023-04-24 18:54:39,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/f16455e49d9d48828cbda00ef99190e2 2023-04-24 18:54:39,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/c250d3800f9044348ca9bbd4be32d713 2023-04-24 18:54:39,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/25ff0b232e2b4657a30c0cb5575c4479 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/25ff0b232e2b4657a30c0cb5575c4479 2023-04-24 18:54:39,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/94dd4947f2364bf7bf9f04e45056208b 2023-04-24 18:54:39,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/6e92e809844b448fba590e8d15d16a04 2023-04-24 18:54:39,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e6f1a9b221554b2790e23712d6e7c6ea to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e6f1a9b221554b2790e23712d6e7c6ea 2023-04-24 18:54:39,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/b546e0cfddf245e0b0626b39bae90082 2023-04-24 18:54:39,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/9c0bddfb41d4438b951c1d524ad1d15e 2023-04-24 18:54:39,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a00109df74fc46029cb81688875a1d65 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a00109df74fc46029cb81688875a1d65 2023-04-24 18:54:39,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/4fcfe260fd8e480d90bf82d2459807c2 2023-04-24 18:54:39,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/e50efba9323343b4b73c0762b9cb418f 2023-04-24 18:54:39,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/55040097d108472aa2af15796ab305a2 2023-04-24 18:54:39,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/bbb6f3e5fdae4661be62a7b590c99aef 2023-04-24 18:54:39,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fdfecbf00ba44c0ac725b741c16feec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7ec3f389991f4dc5af4e3459ca53160a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/036367661a3f4f4e9aae48fdaafe77f6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/94452acecabb491c94a5b5bfd62be6bf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8700f0e5562845db9787b92302cd14a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/e2995374a4b742279d2ea2e9507d8fdf, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a2aec996128d48acbd25f45a27cdadd1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b] to archive 2023-04-24 18:54:39,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:54:39,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/232f0077eaca4943a27339b295215165 2023-04-24 18:54:39,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7240535e800b40458bca3a776fcf650c 2023-04-24 18:54:39,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/0e871729a77b4cfda789a67ecb07fe41 2023-04-24 18:54:39,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9f690f758d540b4b6fe9cdff565cca5 2023-04-24 18:54:39,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c7a0c1cf4248dd8e9391bba956aa65 2023-04-24 18:54:39,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fdfecbf00ba44c0ac725b741c16feec to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fdfecbf00ba44c0ac725b741c16feec 2023-04-24 18:54:39,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d9b2b2c8db574b3489e6e3f40f88330f 2023-04-24 18:54:39,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/54758161cd124cf29466ce1ad2454f4a 2023-04-24 18:54:39,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7ec3f389991f4dc5af4e3459ca53160a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7ec3f389991f4dc5af4e3459ca53160a 2023-04-24 18:54:39,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/17c78fdfafff4fd288784c5d81cc27cc 2023-04-24 18:54:39,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fc66117c1f2648d99e73ae2b93582e46 2023-04-24 18:54:39,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/036367661a3f4f4e9aae48fdaafe77f6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/036367661a3f4f4e9aae48fdaafe77f6 2023-04-24 18:54:39,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/7956d35ecc064fd49c7ef4e164a57cd4 2023-04-24 18:54:39,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/058b63bc5a954aab83ff3baafb1c9643 2023-04-24 18:54:39,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/94452acecabb491c94a5b5bfd62be6bf to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/94452acecabb491c94a5b5bfd62be6bf 2023-04-24 18:54:39,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/15a4a6b89b4842c88a8440fedc171622 2023-04-24 18:54:39,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8fc2ec9271d04d9ab271cc73d046d6bb 2023-04-24 18:54:39,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8700f0e5562845db9787b92302cd14a4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/8700f0e5562845db9787b92302cd14a4 2023-04-24 18:54:39,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/78c2a58c0f0f4a91bb65eb196f65b11f 2023-04-24 18:54:39,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a77e2dbdf7754b9393fbd54364bb60d5 2023-04-24 18:54:39,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/e2995374a4b742279d2ea2e9507d8fdf to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/e2995374a4b742279d2ea2e9507d8fdf 2023-04-24 18:54:39,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/d3e26246c69a4080988046916285c0cc 2023-04-24 18:54:39,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/51da7196b60a48f1a356ac96f3f34126 2023-04-24 18:54:39,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a2aec996128d48acbd25f45a27cdadd1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a2aec996128d48acbd25f45a27cdadd1 2023-04-24 18:54:39,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/248ed6e5a7eb44279f3b546890829aff 2023-04-24 18:54:39,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/fa70af7965a0493b80ec83d943a1ad07 2023-04-24 18:54:39,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/a94b4d816ad2415eb735696849bb74b4 2023-04-24 18:54:39,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/9c7c492860c04f62abf7e17aafb5560b 2023-04-24 18:54:39,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits/847.seqid, newMaxSeqId=847, maxSeqId=4 2023-04-24 18:54:39,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f. 2023-04-24 18:54:39,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 103750326d6eb6a8536f7e1c9d4b7a6f: 2023-04-24 18:54:39,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:39,344 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=103750326d6eb6a8536f7e1c9d4b7a6f, regionState=CLOSED 2023-04-24 18:54:39,344 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362479344"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362479344"}]},"ts":"1682362479344"} 2023-04-24 18:54:39,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-24 18:54:39,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 103750326d6eb6a8536f7e1c9d4b7a6f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 1.9310 sec 2023-04-24 18:54:39,348 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-24 18:54:39,348 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=103750326d6eb6a8536f7e1c9d4b7a6f, UNASSIGN in 1.9350 sec 2023-04-24 18:54:39,348 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362479348"}]},"ts":"1682362479348"} 2023-04-24 18:54:39,349 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:54:39,350 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:54:39,352 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9490 sec 2023-04-24 18:54:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 18:54:39,511 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-24 18:54:39,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:54:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,514 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-24 18:54:39,514 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,519 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:39,521 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits] 2023-04-24 18:54:39,523 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/2509060d142349f49a23ebf706d99272 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/2509060d142349f49a23ebf706d99272 2023-04-24 18:54:39,524 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/bbea757b52604a5ea119f916f1fdf983 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/A/bbea757b52604a5ea119f916f1fdf983 2023-04-24 18:54:39,526 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/40b61dbe0e7744c1ba879bf5b05f2a28 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/40b61dbe0e7744c1ba879bf5b05f2a28 2023-04-24 18:54:39,527 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a8afc7c3b80449159fe628b68ba73a48 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/B/a8afc7c3b80449159fe628b68ba73a48 2023-04-24 18:54:39,530 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/1da20bd8295b40b7ada2149e9ed8ab63 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/1da20bd8295b40b7ada2149e9ed8ab63 2023-04-24 18:54:39,531 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/46407996925e47dcbeda9073dad24c33 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/C/46407996925e47dcbeda9073dad24c33 2023-04-24 18:54:39,533 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits/847.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f/recovered.edits/847.seqid 2023-04-24 18:54:39,534 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/103750326d6eb6a8536f7e1c9d4b7a6f 2023-04-24 18:54:39,534 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:54:39,535 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:54:39,535 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 18:54:39,538 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424168e8739e23947c2b679c93fc915046c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424168e8739e23947c2b679c93fc915046c 2023-04-24 18:54:39,539 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042426690001db344663a73802c609be35e1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042426690001db344663a73802c609be35e1 2023-04-24 18:54:39,540 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042449f49b7e41dc41ddbeab6978f7383bcd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042449f49b7e41dc41ddbeab6978f7383bcd 2023-04-24 18:54:39,541 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042461e2e9559f264005aa43c165cb863d64 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042461e2e9559f264005aa43c165cb863d64 2023-04-24 18:54:39,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304247320ff37583e4a26ae3e2fecf6f07aed to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304247320ff37583e4a26ae3e2fecf6f07aed 2023-04-24 18:54:39,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b0f6b761fc14998bdcd7fd6187ee4df to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b0f6b761fc14998bdcd7fd6187ee4df 2023-04-24 18:54:39,544 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f0801fba8f34cdd84bdeb651727e78b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f0801fba8f34cdd84bdeb651727e78b 2023-04-24 18:54:39,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f8510126cdf4318a41cd269ae20d4ce to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248f8510126cdf4318a41cd269ae20d4ce 2023-04-24 18:54:39,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498903b5a5ccf49f7bb34a199b0aa7cf0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498903b5a5ccf49f7bb34a199b0aa7cf0 2023-04-24 18:54:39,546 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498a61fe673f64254a94e8ceb9c91286a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042498a61fe673f64254a94e8ceb9c91286a 2023-04-24 18:54:39,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a3050f03abe74c3a98316081b5e77160 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a3050f03abe74c3a98316081b5e77160 2023-04-24 18:54:39,548 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a64bd7b4a0104c6e9c8774ec936887b1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a64bd7b4a0104c6e9c8774ec936887b1 2023-04-24 18:54:39,549 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a9bb756207334a5db84b0f9e5478731f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a9bb756207334a5db84b0f9e5478731f 2023-04-24 18:54:39,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ad4d6ec4ca364a3683e1d58eeb3e0d4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ad4d6ec4ca364a3683e1d58eeb3e0d4a 2023-04-24 18:54:39,551 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c23f15fb17524e2daaea3545384ddf60 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c23f15fb17524e2daaea3545384ddf60 2023-04-24 18:54:39,552 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c90d1bb3d7994005bde5fe34add24f9f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c90d1bb3d7994005bde5fe34add24f9f 2023-04-24 18:54:39,553 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cf712844101c4d028030f58aa60763ef to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cf712844101c4d028030f58aa60763ef 2023-04-24 18:54:39,554 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d3417a57863141f6abf1f97d5de37137 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d3417a57863141f6abf1f97d5de37137 2023-04-24 18:54:39,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e1e974f837fa49f9a3098c9ae0c863bf to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e1e974f837fa49f9a3098c9ae0c863bf 2023-04-24 18:54:39,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e5e14cd4cb6743ad8efc2df8f299bf76 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e5e14cd4cb6743ad8efc2df8f299bf76 2023-04-24 18:54:39,556 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f622cf6d74434d92986c4d78e9e61fb1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f622cf6d74434d92986c4d78e9e61fb1 2023-04-24 18:54:39,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f73c2aa0b76246dfaa86d9d8e5c12769 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f73c2aa0b76246dfaa86d9d8e5c12769 2023-04-24 18:54:39,557 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:54:39,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,561 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:54:39,562 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:54:39,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:54:39,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362479563"}]},"ts":"9223372036854775807"} 2023-04-24 18:54:39,565 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:54:39,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 103750326d6eb6a8536f7e1c9d4b7a6f, NAME => 'TestAcidGuarantees,,1682362445583.103750326d6eb6a8536f7e1c9d4b7a6f.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:54:39,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:54:39,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362479565"}]},"ts":"9223372036854775807"} 2023-04-24 18:54:39,566 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:54:39,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-24 18:54:39,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-24 18:54:39,615 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-24 18:54:39,627 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 234), OpenFileDescriptor=531 (was 589), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=378 (was 442), ProcessCount=170 (was 170), AvailableMemoryMB=4259 (was 4487) 2023-04-24 18:54:39,639 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=378, ProcessCount=170, AvailableMemoryMB=4258 2023-04-24 18:54:39,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:54:39,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:54:39,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:54:39,644 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:54:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-24 18:54:39,645 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:54:39,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:39,646 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:54:39,646 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f empty. 2023-04-24 18:54:39,647 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:54:39,647 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:54:39,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:39,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:40,055 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:54:40,057 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => edaa58be4d671676fa0749811078529f, NAME => 'TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:54:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:40,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:54:40,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing edaa58be4d671676fa0749811078529f, disabling compactions & flushes 2023-04-24 18:54:40,462 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. after waiting 0 ms 2023-04-24 18:54:40,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,462 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,463 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:40,464 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:54:40,465 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362480465"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362480465"}]},"ts":"1682362480465"} 2023-04-24 18:54:40,466 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:54:40,467 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:54:40,467 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362480467"}]},"ts":"1682362480467"} 2023-04-24 18:54:40,468 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:54:40,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, ASSIGN}] 2023-04-24 18:54:40,473 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, ASSIGN 2023-04-24 18:54:40,473 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:54:40,624 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=edaa58be4d671676fa0749811078529f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:40,625 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362480624"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362480624"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362480624"}]},"ts":"1682362480624"} 2023-04-24 18:54:40,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:54:40,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:40,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => edaa58be4d671676fa0749811078529f, NAME => 'TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:54:40,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:54:40,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,784 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,785 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A 2023-04-24 18:54:40,785 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A 2023-04-24 18:54:40,785 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:40,785 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region edaa58be4d671676fa0749811078529f columnFamilyName A 2023-04-24 18:54:40,786 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(310): Store=edaa58be4d671676fa0749811078529f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:40,786 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,787 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B 2023-04-24 18:54:40,787 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B 2023-04-24 18:54:40,787 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:40,788 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region edaa58be4d671676fa0749811078529f columnFamilyName B 2023-04-24 18:54:40,788 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(310): Store=edaa58be4d671676fa0749811078529f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:40,788 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,789 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C 2023-04-24 18:54:40,789 DEBUG [StoreOpener-edaa58be4d671676fa0749811078529f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C 2023-04-24 18:54:40,789 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:54:40,789 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region edaa58be4d671676fa0749811078529f columnFamilyName C 2023-04-24 18:54:40,790 INFO [StoreOpener-edaa58be4d671676fa0749811078529f-1] regionserver.HStore(310): Store=edaa58be4d671676fa0749811078529f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:54:40,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:54:40,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for edaa58be4d671676fa0749811078529f 2023-04-24 18:54:40,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:54:40,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened edaa58be4d671676fa0749811078529f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59315081, jitterRate=-0.11613641679286957}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:54:40,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:40,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., pid=30, masterSystemTime=1682362480777 2023-04-24 18:54:40,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:40,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=edaa58be4d671676fa0749811078529f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:40,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362480798"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362480798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362480798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362480798"}]},"ts":"1682362480798"} 2023-04-24 18:54:40,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-24 18:54:40,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 173 msec 2023-04-24 18:54:40,801 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-24 18:54:40,801 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, ASSIGN in 329 msec 2023-04-24 18:54:40,801 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:54:40,802 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362480802"}]},"ts":"1682362480802"} 2023-04-24 18:54:40,802 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:54:40,804 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:54:40,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-24 18:54:41,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 18:54:41,749 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-24 18:54:41,750 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,755 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,756 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,757 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,758 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:54:41,760 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:54:41,761 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,766 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,766 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,770 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,770 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,773 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,774 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,777 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,777 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,782 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,782 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,785 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,786 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,790 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,790 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,794 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,795 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,802 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,802 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:54:41,806 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:54:41,810 DEBUG [hconnection-0x1bd29722-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,810 DEBUG [hconnection-0x6cc8ef11-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,810 DEBUG [hconnection-0x48e1d00a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,811 DEBUG [hconnection-0x62fdc6b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,812 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:41,812 DEBUG [hconnection-0x6f3bcdf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,813 DEBUG [hconnection-0x48df7f91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,816 DEBUG [hconnection-0x16ffdf38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,819 DEBUG [hconnection-0x54590256-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,821 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,819 DEBUG [hconnection-0x6fdd3db0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,823 DEBUG [hconnection-0x240f7ea8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:54:41,825 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,825 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,825 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,825 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,826 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,826 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41418, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,828 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,828 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:54:41,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:41,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:41,831 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:41,831 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:41,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:41,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:41,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:41,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:41,837 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:41,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:41,838 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:41,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:41,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:41,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:41,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:41,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:41,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:41,846 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:41,846 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:41,846 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:41,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:41,868 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/cc3dbd55e8d94c08af9700fa5c56685e 2023-04-24 18:54:41,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362541870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362481865, max timestamp=1682362481873Num uniques 859; 2023-04-24 18:54:41,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:41,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362481865, max timestamp=1682362481873Num uniques 859; 2023-04-24 18:54:41,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:41,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362481865, max timestamp=1682362481873Num uniques 859; 2023-04-24 18:54:41,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:54:41,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362541896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362541898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362541898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362541898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,918 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/90ce0729862e4b20a0d6a5a6de00e08a 2023-04-24 18:54:41,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:41,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362541977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:41,986 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/782b50b280ac49fa91f6420ecbe5767a 2023-04-24 18:54:41,992 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/cc3dbd55e8d94c08af9700fa5c56685e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e 2023-04-24 18:54:41,998 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:54:41,999 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/90ce0729862e4b20a0d6a5a6de00e08a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a 2023-04-24 18:54:42,003 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:54:42,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362542000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,004 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/782b50b280ac49fa91f6420ecbe5767a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a 2023-04-24 18:54:42,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362542002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362542003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362542005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,007 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:54:42,008 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for edaa58be4d671676fa0749811078529f in 162ms, sequenceid=12, compaction requested=false 2023-04-24 18:54:42,008 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:42,008 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:42,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:42,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,009 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:42,009 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:42,011 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769d382f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:42,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:42,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,022 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:42,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:42,024 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:42,027 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:42,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:42,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:42,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:42,027 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:42,028 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,028 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,028 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,030 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:42,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:42,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:42,035 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:42,035 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,135 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:42,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:42,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:42,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:42,139 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:42,139 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,139 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:42,139 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:42,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,141 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,141 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:42,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:42,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:42,143 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:42,143 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,143 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:42,143 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-24 18:54:42,143 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:42,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:42,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:42,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,158 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/5702ef2b1d69483da1d06733e40c0c40 2023-04-24 18:54:42,174 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/334b0511bffe4bb8af05a3a5baee7da5 2023-04-24 18:54:42,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:42,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:42,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362542195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362542204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362542206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362542206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362542207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362542299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362542502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362542508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362542509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362542509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362542510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,584 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/e038056170154796aa01517790226a6b 2023-04-24 18:54:42,589 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/5702ef2b1d69483da1d06733e40c0c40 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40 2023-04-24 18:54:42,593 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:54:42,594 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/334b0511bffe4bb8af05a3a5baee7da5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5 2023-04-24 18:54:42,598 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:54:42,598 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/e038056170154796aa01517790226a6b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b 2023-04-24 18:54:42,602 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:54:42,603 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for edaa58be4d671676fa0749811078529f in 460ms, sequenceid=48, compaction requested=false 2023-04-24 18:54:42,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:42,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:42,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:42,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366728f4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:42,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:42,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:42,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:42,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:42,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:42,622 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:42,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:42,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:42,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:42,622 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:42,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:42,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:42,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,633 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:42,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,633 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:42,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,734 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:42,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:42,738 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:42,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:42,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:42,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:42,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:42,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:42,742 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:54:42,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:42,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,743 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:42,743 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,743 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:42,743 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,756 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/6b7a1e3db10d4020aed5d7de70470a49 2023-04-24 18:54:42,770 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/43abe2712c374adaa23cf0895ba57f06 2023-04-24 18:54:42,783 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/ec9283c689e142b2b4a094c055e53e56 2023-04-24 18:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/6b7a1e3db10d4020aed5d7de70470a49 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49 2023-04-24 18:54:42,800 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49, entries=150, sequenceid=57, filesize=11.6 K 2023-04-24 18:54:42,801 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/43abe2712c374adaa23cf0895ba57f06 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06 2023-04-24 18:54:42,806 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06, entries=150, sequenceid=57, filesize=11.6 K 2023-04-24 18:54:42,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:42,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:42,809 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/ec9283c689e142b2b4a094c055e53e56 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56 2023-04-24 18:54:42,814 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56, entries=150, sequenceid=57, filesize=11.6 K 2023-04-24 18:54:42,815 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=20.13 KB/20610 for edaa58be4d671676fa0749811078529f in 73ms, sequenceid=57, compaction requested=true 2023-04-24 18:54:42,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:42,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:42,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:42,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda7025[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:42,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:42,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:42,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:42,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:42,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:42,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:42,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:42,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:42,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:42,828 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,828 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,828 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,828 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:42,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:42,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:42,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:42,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:42,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:42,837 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:42,837 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:42,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/d80cb0c8f76c4bf3ba2e045f1eebdc56 2023-04-24 18:54:42,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/b61e569e47be4a16ac22183f19c58155 2023-04-24 18:54:42,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362482887, max timestamp=1682362482902Num uniques 859; 2023-04-24 18:54:42,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362482887, max timestamp=1682362482902Num uniques 859; 2023-04-24 18:54:42,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:42,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362482887, max timestamp=1682362482902Num uniques 859; 2023-04-24 18:54:42,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:42,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:42,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/da06b615782e47cfba820db35dd17c59 2023-04-24 18:54:42,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/d80cb0c8f76c4bf3ba2e045f1eebdc56 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56 2023-04-24 18:54:42,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56, entries=200, sequenceid=67, filesize=13.9 K 2023-04-24 18:54:42,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/b61e569e47be4a16ac22183f19c58155 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155 2023-04-24 18:54:42,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155, entries=150, sequenceid=67, filesize=11.6 K 2023-04-24 18:54:42,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/da06b615782e47cfba820db35dd17c59 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59 2023-04-24 18:54:42,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59, entries=150, sequenceid=67, filesize=11.6 K 2023-04-24 18:54:42,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for edaa58be4d671676fa0749811078529f in 95ms, sequenceid=67, compaction requested=true 2023-04-24 18:54:42,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:42,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:42,929 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:42,929 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:42,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:42,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:42,931 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:42,932 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:42,932 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=73.80 KB heapSize=205.69 KB 2023-04-24 18:54:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:42,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:42,932 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,933 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=48.7 K 2023-04-24 18:54:42,934 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:42,934 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cc3dbd55e8d94c08af9700fa5c56685e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362481839 2023-04-24 18:54:42,934 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:42,934 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,934 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=46.4 K 2023-04-24 18:54:42,934 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5702ef2b1d69483da1d06733e40c0c40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362481892 2023-04-24 18:54:42,935 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 90ce0729862e4b20a0d6a5a6de00e08a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362481839 2023-04-24 18:54:42,935 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6b7a1e3db10d4020aed5d7de70470a49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682362482180 2023-04-24 18:54:42,935 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 334b0511bffe4bb8af05a3a5baee7da5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362481892 2023-04-24 18:54:42,936 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d80cb0c8f76c4bf3ba2e045f1eebdc56, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:42,936 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 43abe2712c374adaa23cf0895ba57f06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682362482180 2023-04-24 18:54:42,936 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b61e569e47be4a16ac22183f19c58155, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:42,937 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:42,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:42,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:42,950 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:42,953 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:42,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:42,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:42,956 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:42,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:42,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:42,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:42,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:42,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:42,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:42,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/9cf55328d6af4d7bb98c99984d3f01b8 2023-04-24 18:54:42,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:42,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:42,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:42,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:42,976 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:42,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:42,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:42,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3442e0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:42,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:42,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:42,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:42,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:43,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:43,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:43,003 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:43,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:43,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:43,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:43,003 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:43,004 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,004 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:43,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,011 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,011 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,011 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,011 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,011 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:43,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362543009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362543012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362543012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362543012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,111 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:43,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:43,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:43,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:43,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:43,115 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:43,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:43,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:43,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,117 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,117 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:43,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:43,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:43,119 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,119 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:43,119 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:43,119 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:43,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547747e9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:43,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:43,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:43,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:43,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:43,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:43,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:43,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,127 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:43,130 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:43,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:43,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:43,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:43,130 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:43,133 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,133 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,133 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,134 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,137 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:43,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:43,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,138 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,138 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:43,138 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:43,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,238 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:43,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:43,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:43,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:43,243 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:43,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:43,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:43,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:43,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:43,247 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,247 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:43,247 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:43,247 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bd8f51b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:43,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:43,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,252 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:43,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:43,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:43,260 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:43,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:43,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:43,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:43,260 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:43,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,262 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,266 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:43,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:43,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,266 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:43,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,367 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:43,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:43,370 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:43,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:43,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:43,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:43,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:43,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:43,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:43,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:43,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742cfd1d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:43,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:43,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,380 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/39c5867437664940ad863d3256e8ef89 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/39c5867437664940ad863d3256e8ef89 2023-04-24 18:54:43,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:43,386 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into 39c5867437664940ad863d3256e8ef89(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:43,387 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:43,387 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=12, startTime=1682362482929; duration=0sec 2023-04-24 18:54:43,387 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:43,387 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:43,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,390 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:43,390 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:43,390 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,390 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=46.4 K 2023-04-24 18:54:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,391 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 782b50b280ac49fa91f6420ecbe5767a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362481839 2023-04-24 18:54:43,391 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e038056170154796aa01517790226a6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362481892 2023-04-24 18:54:43,392 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ec9283c689e142b2b4a094c055e53e56, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682362482180 2023-04-24 18:54:43,392 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting da06b615782e47cfba820db35dd17c59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:43,393 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/9196e81d175f4b778c77ce46ac8983ea as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9196e81d175f4b778c77ce46ac8983ea 2023-04-24 18:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:43,394 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:43,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:43,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:43,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:43,394 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:43,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:43,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:43,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,401 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 9196e81d175f4b778c77ce46ac8983ea(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:43,401 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:43,401 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=12, startTime=1682362482929; duration=0sec 2023-04-24 18:54:43,401 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:43,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:43,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:43,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:43,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,403 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:43,403 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:43,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:43,405 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:43,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/e0ab370c107b4a7e99306f33881a3bae 2023-04-24 18:54:43,430 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/d43afcc4267e49a2adc49d7d32b6f0c5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/d43afcc4267e49a2adc49d7d32b6f0c5 2023-04-24 18:54:43,439 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into d43afcc4267e49a2adc49d7d32b6f0c5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:43,439 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:43,439 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=12, startTime=1682362482930; duration=0sec 2023-04-24 18:54:43,439 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:43,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/296ba31471c24558b77bce758974bbb1 2023-04-24 18:54:43,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/9cf55328d6af4d7bb98c99984d3f01b8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8 2023-04-24 18:54:43,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8, entries=200, sequenceid=95, filesize=13.9 K 2023-04-24 18:54:43,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/e0ab370c107b4a7e99306f33881a3bae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae 2023-04-24 18:54:43,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae, entries=150, sequenceid=95, filesize=11.6 K 2023-04-24 18:54:43,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/296ba31471c24558b77bce758974bbb1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1 2023-04-24 18:54:43,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1, entries=150, sequenceid=95, filesize=11.6 K 2023-04-24 18:54:43,464 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.64 KB/210576, currentSize=107.34 KB/109920 for edaa58be4d671676fa0749811078529f in 532ms, sequenceid=95, compaction requested=false 2023-04-24 18:54:43,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:43,503 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:43,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:43,506 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:43,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:43,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:43,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:43,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:43,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:43,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:43,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:43,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:43,510 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:43,511 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:43,511 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:43,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:43,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:43,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:43,922 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/1ae100ecfe9846b5b6eaf3cec378c4a6 2023-04-24 18:54:43,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362543950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362544017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362544017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362544018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362544024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362544254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,332 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/11b35911f79d4ec39ff7c85e55bbf466 2023-04-24 18:54:44,348 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/632720fcf3fd4eb88662d54aa6e9ffcb 2023-04-24 18:54:44,352 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/1ae100ecfe9846b5b6eaf3cec378c4a6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6 2023-04-24 18:54:44,356 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6, entries=150, sequenceid=117, filesize=11.6 K 2023-04-24 18:54:44,357 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/11b35911f79d4ec39ff7c85e55bbf466 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466 2023-04-24 18:54:44,360 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466, entries=150, sequenceid=117, filesize=11.6 K 2023-04-24 18:54:44,361 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/632720fcf3fd4eb88662d54aa6e9ffcb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb 2023-04-24 18:54:44,370 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb, entries=150, sequenceid=117, filesize=11.6 K 2023-04-24 18:54:44,371 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for edaa58be4d671676fa0749811078529f in 860ms, sequenceid=117, compaction requested=true 2023-04-24 18:54:44,371 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:44,371 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:44,371 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:44,371 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:44,371 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:44,372 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:44,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,373 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:44,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:44,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d66823a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:44,376 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:44,376 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,377 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:44,377 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:44,377 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:44,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:44,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:44,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:44,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:44,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:44,382 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:44,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:44,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:44,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:44,383 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:44,386 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,386 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,386 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:44,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,387 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:44,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,391 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:44,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:44,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:44,391 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,392 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:44,392 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:44,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:44,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:44,492 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:44,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:44,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:44,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:44,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:44,495 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:44,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:44,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:44,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:44,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:44,497 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:44,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:44,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:44,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:44,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:44,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:44,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:44,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:44,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:44,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:44,499 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:54:44,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:44,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:44,499 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:44,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:44,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:44,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:44,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:44,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:44,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362544808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:44,882 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:54:44,907 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/203a805f187d4e859290e42b940de087 2023-04-24 18:54:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362544912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:45,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362545115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,317 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/fe0722e88d5440f3b2a06d83f172f8f7 2023-04-24 18:54:45,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41358 deadline: 1682362545419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,728 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/6a55cf1a8b7348bfbcf6492e3a6b24ae 2023-04-24 18:54:45,732 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/203a805f187d4e859290e42b940de087 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087 2023-04-24 18:54:45,737 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087, entries=150, sequenceid=130, filesize=11.7 K 2023-04-24 18:54:45,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/fe0722e88d5440f3b2a06d83f172f8f7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7 2023-04-24 18:54:45,741 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7, entries=150, sequenceid=130, filesize=11.7 K 2023-04-24 18:54:45,742 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/6a55cf1a8b7348bfbcf6492e3a6b24ae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae 2023-04-24 18:54:45,745 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae, entries=150, sequenceid=130, filesize=11.7 K 2023-04-24 18:54:45,746 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for edaa58be4d671676fa0749811078529f in 1247ms, sequenceid=130, compaction requested=true 2023-04-24 18:54:45,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:45,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7340d670[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:45,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:45,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:45,752 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:45,752 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:45,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:45,755 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:45,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:45,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:45,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:45,755 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:45,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:45,757 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,757 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:45,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,758 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,760 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:45,760 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:45,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:45,760 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,760 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:45,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:45,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:45,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:45,761 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:45,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:45,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:45,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:45,761 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:45,761 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:45,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:45,861 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:45,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:45,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:45,864 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:45,864 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:45,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:45,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:45,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:45,867 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:45,867 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:45,868 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:45,868 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:45,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:45,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:45,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:45,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:45,870 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:45,870 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:45,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:45,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362545952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362546029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,032 DEBUG [Thread-1654] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:46,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362546034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,036 DEBUG [Thread-1658] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362546037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,039 DEBUG [Thread-1662] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:46,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362546038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,041 DEBUG [Thread-1660] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:46,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362546055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362546258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,285 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/8f3de500995f47a9aba99250f0b76508 2023-04-24 18:54:46,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:46,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362546562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:46,704 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/23c77bbd96994d219d4039f6e324de7b 2023-04-24 18:54:47,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362547066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,117 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/7bf7777bc5bc491c9b1aa66b6cb1a009 2023-04-24 18:54:47,122 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/8f3de500995f47a9aba99250f0b76508 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508 2023-04-24 18:54:47,126 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508, entries=150, sequenceid=149, filesize=11.8 K 2023-04-24 18:54:47,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/23c77bbd96994d219d4039f6e324de7b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b 2023-04-24 18:54:47,130 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b, entries=150, sequenceid=149, filesize=11.8 K 2023-04-24 18:54:47,131 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/7bf7777bc5bc491c9b1aa66b6cb1a009 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009 2023-04-24 18:54:47,135 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009, entries=150, sequenceid=149, filesize=11.8 K 2023-04-24 18:54:47,135 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for edaa58be4d671676fa0749811078529f in 1265ms, sequenceid=149, compaction requested=true 2023-04-24 18:54:47,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:47,135 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:47,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fac952e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:47,139 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:47,139 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,140 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:47,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,140 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:47,141 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:47,144 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:47,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:47,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:47,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:47,144 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:47,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:47,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:47,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:47,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:47,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:47,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:47,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:47,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:47,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:47,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:47,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:47,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,154 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:47,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:47,154 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:47,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:47,254 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:47,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:47,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:47,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:47,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:47,257 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:47,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:47,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:47,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:47,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:47,259 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:47,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:47,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:47,260 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:47,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:47,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:47,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:47,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:47,261 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:47,261 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:47,261 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:47,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:47,670 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/92047f3618414f3eaf38e2977558566c 2023-04-24 18:54:48,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:48,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:48,081 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/98901882e417452c9d02757982652bca 2023-04-24 18:54:48,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362548142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362548245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362548447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,496 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/8d02254218ed4ed6876ff453c5c7a653 2023-04-24 18:54:48,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/92047f3618414f3eaf38e2977558566c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c 2023-04-24 18:54:48,504 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c, entries=150, sequenceid=162, filesize=11.8 K 2023-04-24 18:54:48,505 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/98901882e417452c9d02757982652bca as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca 2023-04-24 18:54:48,508 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca, entries=150, sequenceid=162, filesize=11.8 K 2023-04-24 18:54:48,509 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/8d02254218ed4ed6876ff453c5c7a653 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653 2023-04-24 18:54:48,512 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653, entries=150, sequenceid=162, filesize=11.8 K 2023-04-24 18:54:48,513 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for edaa58be4d671676fa0749811078529f in 1252ms, sequenceid=162, compaction requested=true 2023-04-24 18:54:48,513 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:48,513 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:48,515 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfb49d3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:48,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:48,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,519 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:48,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:48,520 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:48,523 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:48,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:48,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:48,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:48,523 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:48,524 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,524 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,524 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:48,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:48,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:48,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:48,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:48,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:48,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:48,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:48,528 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:48,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:48,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:48,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:48,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:48,529 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:48,529 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:48,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:48,629 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:48,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:48,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:48,632 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:48,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:48,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:48,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:48,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:48,635 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:48,635 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:48,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:48,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:48,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:48,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:48,637 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:48,638 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:48,638 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:48,753 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:48,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:48,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362548782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:48,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:48,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362548886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,045 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/cfcb20933aa5415a99c6d94db511ff14 2023-04-24 18:54:49,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362549088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:49,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362549393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,458 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/0e9908959fa44b218684e3d9396a3b14 2023-04-24 18:54:49,468 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/a5102e560be34f079869b366d336373b 2023-04-24 18:54:49,472 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/cfcb20933aa5415a99c6d94db511ff14 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14 2023-04-24 18:54:49,476 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14, entries=150, sequenceid=181, filesize=11.8 K 2023-04-24 18:54:49,477 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/0e9908959fa44b218684e3d9396a3b14 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14 2023-04-24 18:54:49,481 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14, entries=150, sequenceid=181, filesize=11.8 K 2023-04-24 18:54:49,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/a5102e560be34f079869b366d336373b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b 2023-04-24 18:54:49,484 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b, entries=150, sequenceid=181, filesize=11.8 K 2023-04-24 18:54:49,485 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for edaa58be4d671676fa0749811078529f in 848ms, sequenceid=181, compaction requested=true 2023-04-24 18:54:49,485 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:49,485 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,486 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:49,486 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:49,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:49,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55257324[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:49,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:49,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:49,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:49,492 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:49,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:49,494 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:49,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:49,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:49,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:49,494 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:49,496 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,496 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,496 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:49,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,501 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:49,501 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,601 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:49,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:49,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:49,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:49,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:49,605 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:49,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:49,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:49,606 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,606 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:49,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:49,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:49,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:49,609 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,609 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:49,609 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:54:49,609 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:49,609 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:49,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:49,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,617 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2ee2f6a732a142d898dc33d0a78383ce 2023-04-24 18:54:49,627 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/636fde17bc484cdf89bb865b949d41a4 2023-04-24 18:54:49,636 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/9cf11d9044214714849b1341e56a7d01 2023-04-24 18:54:49,641 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2ee2f6a732a142d898dc33d0a78383ce as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce 2023-04-24 18:54:49,645 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce, entries=150, sequenceid=194, filesize=11.8 K 2023-04-24 18:54:49,646 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/636fde17bc484cdf89bb865b949d41a4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4 2023-04-24 18:54:49,649 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4, entries=150, sequenceid=194, filesize=11.8 K 2023-04-24 18:54:49,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/9cf11d9044214714849b1341e56a7d01 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01 2023-04-24 18:54:49,656 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01, entries=150, sequenceid=194, filesize=11.8 K 2023-04-24 18:54:49,657 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for edaa58be4d671676fa0749811078529f in 48ms, sequenceid=194, compaction requested=true 2023-04-24 18:54:49,657 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:49,657 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:49,659 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,659 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2023dca6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:49,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:49,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:49,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:49,663 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,663 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:49,664 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:49,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:49,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:49,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:49,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:49,667 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:49,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,676 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:49,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:49,676 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:49,676 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,776 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:49,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:49,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:49,780 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,781 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,782 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,782 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:49,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:49,784 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,784 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:49,784 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:49,784 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:49,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,786 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d2869[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:49,788 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:49,788 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,790 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:49,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:49,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:49,795 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:49,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:49,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:49,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:49,795 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:49,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:49,797 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,799 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:49,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:49,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,801 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:49,802 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:49,902 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:49,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:49,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:49,907 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:49,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:49,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:49,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:49,909 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:49,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:49,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:49,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:49,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:49,912 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:49,912 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:49,921 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0a931a907f6a47679f704ff83f3b703c 2023-04-24 18:54:49,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:49,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362489956, max timestamp=1682362489991Num uniques 859; 2023-04-24 18:54:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362489956, max timestamp=1682362489991Num uniques 859; 2023-04-24 18:54:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:49,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362489956, max timestamp=1682362489991Num uniques 859; 2023-04-24 18:54:49,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:50,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362490041, max timestamp=1682362490058Num uniques 1018; 2023-04-24 18:54:50,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362490041, max timestamp=1682362490058Num uniques 1018; 2023-04-24 18:54:50,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:50,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:50,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362490041, max timestamp=1682362490058Num uniques 1018; 2023-04-24 18:54:50,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:54:50,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362490072, max timestamp=1682362490082Num uniques 1077; 2023-04-24 18:54:50,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:54:50,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362490072, max timestamp=1682362490082Num uniques 1077; 2023-04-24 18:54:50,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362490072, max timestamp=1682362490082Num uniques 1077; 2023-04-24 18:54:50,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:54:50,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 18:54:50,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362490090, max timestamp=1682362490103Num uniques 1086; 2023-04-24 18:54:50,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:54:50,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362490090, max timestamp=1682362490103Num uniques 1086; 2023-04-24 18:54:50,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:54:50,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682362490090, max timestamp=1682362490103Num uniques 1086; 2023-04-24 18:54:50,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-24 18:54:50,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682362490111, max timestamp=1682362490124Num uniques 1195; 2023-04-24 18:54:50,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-24 18:54:50,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682362490111, max timestamp=1682362490124Num uniques 1195; 2023-04-24 18:54:50,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-24 18:54:50,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682362490111, max timestamp=1682362490124Num uniques 1195; 2023-04-24 18:54:50,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-24 18:54:50,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362550143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362550145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362550145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362550146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362550159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362550260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362550262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362550262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362550262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362550264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,331 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/78ac6fab66bb429886e2f490f24fd21a 2023-04-24 18:54:50,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362550463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362550465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362550465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362550467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362550468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,741 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/fb37d69e6c3f4e6e8f99376c6b521d9b 2023-04-24 18:54:50,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0a931a907f6a47679f704ff83f3b703c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c 2023-04-24 18:54:50,750 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c, entries=150, sequenceid=202, filesize=11.8 K 2023-04-24 18:54:50,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/78ac6fab66bb429886e2f490f24fd21a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a 2023-04-24 18:54:50,757 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a, entries=150, sequenceid=202, filesize=11.8 K 2023-04-24 18:54:50,757 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/fb37d69e6c3f4e6e8f99376c6b521d9b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b 2023-04-24 18:54:50,762 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b, entries=150, sequenceid=202, filesize=11.8 K 2023-04-24 18:54:50,763 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for edaa58be4d671676fa0749811078529f in 851ms, sequenceid=202, compaction requested=true 2023-04-24 18:54:50,763 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:50,763 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:50,768 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dbe590e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:50,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:50,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,771 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:50,771 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,771 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:50,773 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:50,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:50,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=167.72 KB heapSize=467.86 KB 2023-04-24 18:54:50,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:50,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:50,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:50,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:50,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:50,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362550773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362550775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:50,779 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:50,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:50,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:50,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:50,779 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:50,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362550775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362550776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,782 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:50,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362550777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,784 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:50,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:50,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:50,786 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,787 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,787 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:50,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:50,787 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:50,787 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:50,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362550880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362550883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362550883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,887 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:50,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362550884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:50,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:50,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:50,890 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:50,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:50,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:50,890 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:50,891 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,891 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,892 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,892 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:50,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:50,892 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,892 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:50,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:50,894 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:50,894 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:50,894 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:50,894 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:50,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:50,895 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737ca7ce[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:50,897 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:50,897 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:50,900 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,900 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:50,902 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:50,904 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:50,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:50,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:50,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:50,904 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:50,906 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,906 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,906 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:50,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:50,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:50,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:50,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:50,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:50,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:50,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:50,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:50,911 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:50,911 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:50,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,011 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,015 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,015 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,020 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,021 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,022 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,022 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,022 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,026 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0493eb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,030 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,033 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,033 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,039 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,043 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,043 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362551085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362551086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362551086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362551088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,143 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,146 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,148 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa89442[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,160 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,160 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,162 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,166 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,166 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/1b53b298d98f41e8b7c668fe9934f038 2023-04-24 18:54:51,266 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,269 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,269 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,272 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,274 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,274 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,274 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,274 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a1da90[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,279 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,279 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,280 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,283 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,283 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41346 deadline: 1682362551282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,289 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,290 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362551388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,390 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362551388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362551389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,393 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,393 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,393 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,393 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,394 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,395 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362551393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,397 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,397 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,397 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,397 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f5519d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,406 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,407 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,412 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,412 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:51,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,512 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,516 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,520 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,522 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f071d63[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,526 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,526 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,529 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,529 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,531 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,531 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,531 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,534 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,538 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,539 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,539 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ead69b87e949454db4975eb519cc867a 2023-04-24 18:54:51,639 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,642 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,644 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,646 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,646 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,646 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,646 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,648 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c68245[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,653 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,655 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,655 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,655 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,655 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,655 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,662 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,662 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,762 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,765 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,766 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,766 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,768 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,768 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,769 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,769 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,770 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,770 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,770 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,770 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,772 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a4da517[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,774 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,774 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,783 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,783 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,784 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,784 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,785 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,785 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:51,790 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,790 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,790 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,790 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,890 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:51,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:51,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:51,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:51,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,893 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:51,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362551892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:51,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:51,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362551892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,894 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,894 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,895 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,895 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:51,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:51,897 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,897 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:51,897 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:51,897 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:51,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362551895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,899 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,899 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,899 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7678727a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:51,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:51,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,902 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362551901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:51,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:51,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:51,908 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:51,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:51,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:51,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:51,908 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:51,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:51,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:51,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:51,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:51,914 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:51,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:51,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:51,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:51,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:51,915 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:51,915 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:51,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:52,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/b8ed4a12025d4eff96983879e1e88e9e 2023-04-24 18:54:52,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/1b53b298d98f41e8b7c668fe9934f038 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038 2023-04-24 18:54:52,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038, entries=200, sequenceid=309, filesize=14.3 K 2023-04-24 18:54:52,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ead69b87e949454db4975eb519cc867a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a 2023-04-24 18:54:52,015 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:52,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:52,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a, entries=150, sequenceid=309, filesize=11.9 K 2023-04-24 18:54:52,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/b8ed4a12025d4eff96983879e1e88e9e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e 2023-04-24 18:54:52,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:52,018 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:52,020 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:52,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:52,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e, entries=150, sequenceid=309, filesize=11.9 K 2023-04-24 18:54:52,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~467.81 KB/479040, currentSize=20.13 KB/20610 for edaa58be4d671676fa0749811078529f in 1249ms, sequenceid=309, compaction requested=true 2023-04-24 18:54:52,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:52,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:52,022 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-24 18:54:52,022 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-24 18:54:52,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:52,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:52,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:52,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:52,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:52,025 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:52,025 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:52,025 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:52,026 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 124878 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-24 18:54:52,026 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:52,026 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:52,026 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:52,026 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120098 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-24 18:54:52,026 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:52,026 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:52,026 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/39c5867437664940ad863d3256e8ef89, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=117.3 K 2023-04-24 18:54:52,026 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9196e81d175f4b778c77ce46ac8983ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=122.0 K 2023-04-24 18:54:52,027 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 39c5867437664940ad863d3256e8ef89, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:52,027 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9196e81d175f4b778c77ce46ac8983ea, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:52,027 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9cf55328d6af4d7bb98c99984d3f01b8, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362482906 2023-04-24 18:54:52,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e0ab370c107b4a7e99306f33881a3bae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362482906 2023-04-24 18:54:52,028 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1ae100ecfe9846b5b6eaf3cec378c4a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682362482986 2023-04-24 18:54:52,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 11b35911f79d4ec39ff7c85e55bbf466, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682362482986 2023-04-24 18:54:52,028 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 203a805f187d4e859290e42b940de087, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682362483627 2023-04-24 18:54:52,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fe0722e88d5440f3b2a06d83f172f8f7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682362483627 2023-04-24 18:54:52,029 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8f3de500995f47a9aba99250f0b76508, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682362484766 2023-04-24 18:54:52,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 23c77bbd96994d219d4039f6e324de7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682362484766 2023-04-24 18:54:52,029 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 92047f3618414f3eaf38e2977558566c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362485945 2023-04-24 18:54:52,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 98901882e417452c9d02757982652bca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362485945 2023-04-24 18:54:52,029 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cfcb20933aa5415a99c6d94db511ff14, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682362488118 2023-04-24 18:54:52,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0e9908959fa44b218684e3d9396a3b14, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682362488118 2023-04-24 18:54:52,030 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2ee2f6a732a142d898dc33d0a78383ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682362488758 2023-04-24 18:54:52,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 636fde17bc484cdf89bb865b949d41a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682362488758 2023-04-24 18:54:52,030 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0a931a907f6a47679f704ff83f3b703c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682362489904 2023-04-24 18:54:52,031 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1b53b298d98f41e8b7c668fe9934f038, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490130 2023-04-24 18:54:52,031 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 78ac6fab66bb429886e2f490f24fd21a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682362489904 2023-04-24 18:54:52,031 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ead69b87e949454db4975eb519cc867a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490145 2023-04-24 18:54:52,034 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/3bccc67c8cba4f7d9c7d6279e130b30a 2023-04-24 18:54:52,053 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#248 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:52,057 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#249 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:52,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:52,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362492342, max timestamp=1682362492370Num uniques 859; 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362492342, max timestamp=1682362492370Num uniques 859; 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362492342, max timestamp=1682362492370Num uniques 859; 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:52,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:52,460 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/87728e5103794cd887cf4d6a6445a62a 2023-04-24 18:54:52,465 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/652dcc2825f84b69a2d6c168c14bc38f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/652dcc2825f84b69a2d6c168c14bc38f 2023-04-24 18:54:52,471 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 652dcc2825f84b69a2d6c168c14bc38f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:52,472 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:52,472 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=6, startTime=1682362492022; duration=0sec 2023-04-24 18:54:52,472 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:52,472 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362492427, max timestamp=1682362492469Num uniques 1018; 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362492427, max timestamp=1682362492469Num uniques 1018; 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362492427, max timestamp=1682362492469Num uniques 1018; 2023-04-24 18:54:52,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:52,473 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/a6d7add99d9242daac5e09661e708c48 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/a6d7add99d9242daac5e09661e708c48 2023-04-24 18:54:52,476 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120098 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-24 18:54:52,476 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:52,476 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:52,476 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/d43afcc4267e49a2adc49d7d32b6f0c5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=117.3 K 2023-04-24 18:54:52,477 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d43afcc4267e49a2adc49d7d32b6f0c5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1682362482822 2023-04-24 18:54:52,477 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 296ba31471c24558b77bce758974bbb1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362482906 2023-04-24 18:54:52,478 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 632720fcf3fd4eb88662d54aa6e9ffcb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682362482986 2023-04-24 18:54:52,478 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into a6d7add99d9242daac5e09661e708c48(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:52,478 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:52,478 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=6, startTime=1682362492022; duration=0sec 2023-04-24 18:54:52,478 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6a55cf1a8b7348bfbcf6492e3a6b24ae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682362483627 2023-04-24 18:54:52,478 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:52,479 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7bf7777bc5bc491c9b1aa66b6cb1a009, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682362484766 2023-04-24 18:54:52,479 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8d02254218ed4ed6876ff453c5c7a653, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362485945 2023-04-24 18:54:52,479 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a5102e560be34f079869b366d336373b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682362488118 2023-04-24 18:54:52,479 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9cf11d9044214714849b1341e56a7d01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682362488758 2023-04-24 18:54:52,480 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fb37d69e6c3f4e6e8f99376c6b521d9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682362489904 2023-04-24 18:54:52,480 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b8ed4a12025d4eff96983879e1e88e9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490145 2023-04-24 18:54:52,495 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:52,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362492527, max timestamp=1682362492541Num uniques 1027; 2023-04-24 18:54:52,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-24 18:54:52,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362492527, max timestamp=1682362492541Num uniques 1027; 2023-04-24 18:54:52,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362492527, max timestamp=1682362492541Num uniques 1027; 2023-04-24 18:54:52,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-24 18:54:52,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-24 18:54:52,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362492588, max timestamp=1682362492603Num uniques 1036; 2023-04-24 18:54:52,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-24 18:54:52,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362492588, max timestamp=1682362492603Num uniques 1036; 2023-04-24 18:54:52,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-24 18:54:52,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362492588, max timestamp=1682362492603Num uniques 1036; 2023-04-24 18:54:52,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-24 18:54:52,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362492659, max timestamp=1682362492669Num uniques 1045; 2023-04-24 18:54:52,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:54:52,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362492659, max timestamp=1682362492669Num uniques 1045; 2023-04-24 18:54:52,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:54:52,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362492659, max timestamp=1682362492669Num uniques 1045; 2023-04-24 18:54:52,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-24 18:54:52,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362492705, max timestamp=1682362492725Num uniques 1054; 2023-04-24 18:54:52,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-24 18:54:52,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362492705, max timestamp=1682362492725Num uniques 1054; 2023-04-24 18:54:52,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-24 18:54:52,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362492705, max timestamp=1682362492725Num uniques 1054; 2023-04-24 18:54:52,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362492765, max timestamp=1682362492783Num uniques 1013; 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362492765, max timestamp=1682362492783Num uniques 1013; 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362492765, max timestamp=1682362492783Num uniques 1013; 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:52,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362492817, max timestamp=1682362492836Num uniques 1022; 2023-04-24 18:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 18:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362492817, max timestamp=1682362492836Num uniques 1022; 2023-04-24 18:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 18:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362492817, max timestamp=1682362492836Num uniques 1022; 2023-04-24 18:54:52,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 18:54:52,875 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/43729b4994774ee492d50c03f273fcac 2023-04-24 18:54:52,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/3bccc67c8cba4f7d9c7d6279e130b30a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a 2023-04-24 18:54:52,883 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a, entries=100, sequenceid=315, filesize=9.5 K 2023-04-24 18:54:52,884 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/87728e5103794cd887cf4d6a6445a62a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a 2023-04-24 18:54:52,890 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a, entries=100, sequenceid=315, filesize=9.5 K 2023-04-24 18:54:52,891 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/43729b4994774ee492d50c03f273fcac as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac 2023-04-24 18:54:52,895 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac, entries=100, sequenceid=315, filesize=9.5 K 2023-04-24 18:54:52,895 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for edaa58be4d671676fa0749811078529f in 870ms, sequenceid=315, compaction requested=false 2023-04-24 18:54:52,896 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:52,896 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:52,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:52,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=134.18 KB heapSize=383.77 KB 2023-04-24 18:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:52,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:52,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:52,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:52,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:52,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412385d3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:52,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:52,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,903 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:52,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:52,905 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/8cfe9bee1b7a4abc87599373197040bd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8cfe9bee1b7a4abc87599373197040bd 2023-04-24 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:52,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,905 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:52,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:52,908 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:52,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:52,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:52,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:52,908 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:52,909 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into 8cfe9bee1b7a4abc87599373197040bd(size=12.4 K), total size for store is 21.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:52,909 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:52,909 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=6, startTime=1682362492023; duration=0sec 2023-04-24 18:54:52,909 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:52,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:52,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:52,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:52,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362552909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:52,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362552911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:52,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362552912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:52,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:52,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:52,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:52,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:52,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:52,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:52,917 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:52,917 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:52,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:52,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362552913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362552914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,017 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362553015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362553015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362553016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,020 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362553018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,021 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362553018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,022 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,024 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,024 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,024 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,024 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,025 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3187f318[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,027 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,028 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,030 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,033 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,033 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,035 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,036 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,041 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,041 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,141 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,144 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,147 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,150 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66358d22[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,168 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,170 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,171 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,171 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362553218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362553219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362553219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362553221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362553222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,271 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,274 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,277 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,279 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ebd83c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,287 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,287 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,294 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,294 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/b560a9fc02f34b3592c36a93bbbeacdc 2023-04-24 18:54:53,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ba548269aea141f19e04c289de26a8df 2023-04-24 18:54:53,394 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,398 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,401 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,403 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebbd28f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,411 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,411 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,415 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,420 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,420 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,520 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362553521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,523 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362553523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362553524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362553525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362553525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,527 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,528 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,528 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,530 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aaef73[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,533 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,533 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,533 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,534 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,534 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,537 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,537 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,540 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,545 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,549 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,549 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,649 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,652 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,654 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,654 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:53,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:53,658 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abc3ed9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:53,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:53,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:53,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:53,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:53,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:53,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:53,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:53,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:53,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,672 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:53,672 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:53,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:53,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:53,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:53,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/5eb48f60a0cd47e4b08f61bdc159a8af 2023-04-24 18:54:53,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/b560a9fc02f34b3592c36a93bbbeacdc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc 2023-04-24 18:54:53,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc, entries=150, sequenceid=475, filesize=11.9 K 2023-04-24 18:54:53,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ba548269aea141f19e04c289de26a8df as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df 2023-04-24 18:54:53,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df, entries=150, sequenceid=475, filesize=11.9 K 2023-04-24 18:54:53,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/5eb48f60a0cd47e4b08f61bdc159a8af as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af 2023-04-24 18:54:53,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af, entries=150, sequenceid=475, filesize=11.9 K 2023-04-24 18:54:53,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~404.34 KB/414048, currentSize=40.25 KB/41220 for edaa58be4d671676fa0749811078529f in 853ms, sequenceid=475, compaction requested=true 2023-04-24 18:54:53,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:53,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:53,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:53,752 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:53,752 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:53,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:53,753 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:53,753 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:53,754 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:53,754 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:53,754 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,754 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,754 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/652dcc2825f84b69a2d6c168c14bc38f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=33.8 K 2023-04-24 18:54:53,754 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/a6d7add99d9242daac5e09661e708c48, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=33.8 K 2023-04-24 18:54:53,754 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 652dcc2825f84b69a2d6c168c14bc38f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490145 2023-04-24 18:54:53,754 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a6d7add99d9242daac5e09661e708c48, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490145 2023-04-24 18:54:53,755 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3bccc67c8cba4f7d9c7d6279e130b30a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682362490775 2023-04-24 18:54:53,755 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 87728e5103794cd887cf4d6a6445a62a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682362490775 2023-04-24 18:54:53,755 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b560a9fc02f34b3592c36a93bbbeacdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:53,755 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ba548269aea141f19e04c289de26a8df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:53,766 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:53,766 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:53,772 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:53,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:53,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:53,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:53,775 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:53,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:53,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:53,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,777 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:53,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:53,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:53,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:53,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:53,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:53,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:53,779 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:54:53,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:53,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:53,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:53,780 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:53,780 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:53,780 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:53,786 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/f799a252b674401ba23ad5261c678d96 2023-04-24 18:54:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:54,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:54,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682362494037, max timestamp=1682362494044Num uniques 859; 2023-04-24 18:54:54,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 18:54:54,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682362494037, max timestamp=1682362494044Num uniques 859; 2023-04-24 18:54:54,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 18:54:54,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682362494037, max timestamp=1682362494044Num uniques 859; 2023-04-24 18:54:54,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 18:54:54,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362554078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362554078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362554079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362554081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362554081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,183 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ada1cd50099040aba493b871ee6596e1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ada1cd50099040aba493b871ee6596e1 2023-04-24 18:54:54,185 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/896194be52fc4672b333f64d49bf674d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/896194be52fc4672b333f64d49bf674d 2023-04-24 18:54:54,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362554185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362554185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362554185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362554186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362554187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,189 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into ada1cd50099040aba493b871ee6596e1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:54,189 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:54,189 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=13, startTime=1682362493752; duration=0sec 2023-04-24 18:54:54,189 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:54,189 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:54,191 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:54,191 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:54,191 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,192 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8cfe9bee1b7a4abc87599373197040bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=33.8 K 2023-04-24 18:54:54,192 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8cfe9bee1b7a4abc87599373197040bd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682362490145 2023-04-24 18:54:54,192 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 43729b4994774ee492d50c03f273fcac, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682362490775 2023-04-24 18:54:54,193 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5eb48f60a0cd47e4b08f61bdc159a8af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:54,193 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 896194be52fc4672b333f64d49bf674d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:54,194 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:54,194 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=13, startTime=1682362493752; duration=0sec 2023-04-24 18:54:54,194 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:54,196 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/3fac3761626a4d7c8f56ad38dbaf4b16 2023-04-24 18:54:54,199 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362554388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362554388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362554389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362554389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362554392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,614 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/020b8a22cc384accb3b737c6b175fa89 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/020b8a22cc384accb3b737c6b175fa89 2023-04-24 18:54:54,618 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/5294c7fb795f44a89a19476f02a7d86d 2023-04-24 18:54:54,618 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into 020b8a22cc384accb3b737c6b175fa89(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:54,618 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:54,618 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=13, startTime=1682362493752; duration=0sec 2023-04-24 18:54:54,618 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:54,622 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/f799a252b674401ba23ad5261c678d96 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96 2023-04-24 18:54:54,625 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96, entries=150, sequenceid=485, filesize=11.9 K 2023-04-24 18:54:54,626 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/3fac3761626a4d7c8f56ad38dbaf4b16 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16 2023-04-24 18:54:54,629 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16, entries=150, sequenceid=485, filesize=11.9 K 2023-04-24 18:54:54,630 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/5294c7fb795f44a89a19476f02a7d86d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d 2023-04-24 18:54:54,634 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d, entries=150, sequenceid=485, filesize=11.9 K 2023-04-24 18:54:54,635 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for edaa58be4d671676fa0749811078529f in 856ms, sequenceid=485, compaction requested=false 2023-04-24 18:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:54,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:54,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,638 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471b358a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:54,639 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:54,639 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:54,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:54,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:54,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,642 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:54,644 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:54,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:54,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:54,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:54,644 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:54,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:54,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:54,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:54,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:54,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,651 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:54,651 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:54,651 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:54,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:54,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:54,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=167.72 KB heapSize=466.55 KB 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:54,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:54,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362554695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362554699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362554720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362554720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362554720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,752 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:54,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:54,755 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:54,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:54,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:54,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:54,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,759 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:54,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:54,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:54,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:54,762 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,762 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:54,762 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:54,762 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:54,764 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,765 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1764b84c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:54,766 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:54,767 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,768 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:54,768 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,768 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:54,768 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:54,770 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:54,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:54,773 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:54,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:54,773 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:54,774 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,774 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,774 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,775 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,781 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,781 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:54,781 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,781 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362554822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362554823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362554823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,881 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:54,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:54,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:54,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:54,885 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:54,885 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:54,885 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:54,885 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:54,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,887 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:54,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:54,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:54,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:54,889 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,889 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:54,889 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:54,889 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:54,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:54,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40621ca4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:54,892 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:54,892 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,893 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,894 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,894 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:54,896 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:54,898 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:54,898 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:54,898 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:54,898 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:54,899 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:54,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:54,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:54,901 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:54,904 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:54,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:54,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:54,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:54,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:54,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:54,904 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:54,904 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:54,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,005 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,008 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,010 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,011 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,011 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,012 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,012 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,012 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:55,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:55,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:55,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:55,014 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:55,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b03b887[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:55,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:55,016 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:55,017 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:55,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:55,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,018 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,018 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,019 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:55,021 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:55,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:55,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:55,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:55,022 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:55,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,023 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,024 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,025 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362555025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362555026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,029 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:55,029 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:55,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362555027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2969f0db926144ce969ab514b2dee43d 2023-04-24 18:54:55,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/f6c75577072e48e9abd3212d0677c809 2023-04-24 18:54:55,129 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,133 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,134 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,134 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,136 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,136 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,136 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,136 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:55,141 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@752a7708[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:55,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:55,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,145 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:55,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:55,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:55,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:55,148 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:55,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:55,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:55,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:55,149 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:55,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:55,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:55,159 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:55,159 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:55,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362555222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362555223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,259 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,263 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,263 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:55,270 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:55,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47be0739[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:55,272 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,273 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:55,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:55,274 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:55,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:55,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:55,277 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:55,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:55,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:55,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:55,277 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:55,280 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,280 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,280 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:55,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,284 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:55,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,285 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:55,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41434 deadline: 1682362555329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41374 deadline: 1682362555329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:55,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41420 deadline: 1682362555332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,385 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,388 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,389 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,391 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:55,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e9b456[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:55,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:55,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,397 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:55,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:55,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:55,402 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:55,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:55,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:55,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:55,402 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:55,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:55,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,408 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:55,409 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:55,509 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,514 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,519 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,519 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,519 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,519 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:55,521 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,521 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/506fd31df0e84ef996bc6f94a9e6e86a 2023-04-24 18:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:55,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d8e1ac[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:55,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:55,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,524 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:55,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,525 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:55,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2969f0db926144ce969ab514b2dee43d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d 2023-04-24 18:54:55,529 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d, entries=250, sequenceid=526, filesize=16.7 K 2023-04-24 18:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/f6c75577072e48e9abd3212d0677c809 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809 2023-04-24 18:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:55,532 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:55,533 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:55,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809, entries=150, sequenceid=526, filesize=11.9 K 2023-04-24 18:54:55,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:55,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/506fd31df0e84ef996bc6f94a9e6e86a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a 2023-04-24 18:54:55,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:55,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,540 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:55,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:55,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:55,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a, entries=150, sequenceid=526, filesize=11.9 K 2023-04-24 18:54:55,541 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:55,541 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:55,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:55,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:55,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~466.50 KB/477696, currentSize=20.13 KB/20610 for edaa58be4d671676fa0749811078529f in 849ms, sequenceid=526, compaction requested=true 2023-04-24 18:54:55,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:55,542 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:55,542 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:55,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:55,543 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42018 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:55,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:55,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/896194be52fc4672b333f64d49bf674d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=41.0 K 2023-04-24 18:54:55,545 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:55,545 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:55,545 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 896194be52fc4672b333f64d49bf674d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:55,546 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ada1cd50099040aba493b871ee6596e1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=36.3 K 2023-04-24 18:54:55,546 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f799a252b674401ba23ad5261c678d96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682362492909 2023-04-24 18:54:55,547 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ada1cd50099040aba493b871ee6596e1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:55,547 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3fac3761626a4d7c8f56ad38dbaf4b16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682362492909 2023-04-24 18:54:55,547 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2969f0db926144ce969ab514b2dee43d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494055 2023-04-24 18:54:55,548 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f6c75577072e48e9abd3212d0677c809, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494077 2023-04-24 18:54:55,561 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:55,569 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:55,580 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/96800baeb6ce40dd96f8bd110f0997f7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/96800baeb6ce40dd96f8bd110f0997f7 2023-04-24 18:54:55,593 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 96800baeb6ce40dd96f8bd110f0997f7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:55,593 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:55,593 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=13, startTime=1682362495542; duration=0sec 2023-04-24 18:54:55,593 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:55,594 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:55,595 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:55,595 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:55,595 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,595 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/020b8a22cc384accb3b737c6b175fa89, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=36.3 K 2023-04-24 18:54:55,596 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 020b8a22cc384accb3b737c6b175fa89, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682362492890 2023-04-24 18:54:55,596 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5294c7fb795f44a89a19476f02a7d86d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1682362492909 2023-04-24 18:54:55,596 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 506fd31df0e84ef996bc6f94a9e6e86a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494077 2023-04-24 18:54:55,604 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#266 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:55,614 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/c3b8d3265de8467fbbef058e4a8e0d34 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c3b8d3265de8467fbbef058e4a8e0d34 2023-04-24 18:54:55,618 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into c3b8d3265de8467fbbef058e4a8e0d34(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:55,618 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:55,618 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=13, startTime=1682362495542; duration=0sec 2023-04-24 18:54:55,619 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:55,641 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:55,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:55,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:55,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:55,645 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:55,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:55,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:55,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,645 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:55,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:55,648 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:55,648 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:55,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:55,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:55,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:55,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:55,649 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:55,649 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:55,649 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-24 18:54:55,649 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:55,649 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:55,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:55,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:55,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:55,650 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:55,657 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/43d6d0b11174466d9b2a87cb7ae27835 2023-04-24 18:54:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:55,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362495838, max timestamp=1682362495848Num uniques 859; 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362495838, max timestamp=1682362495848Num uniques 859; 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362495838, max timestamp=1682362495848Num uniques 859; 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:55,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:55,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362495862, max timestamp=1682362495867Num uniques 1018; 2023-04-24 18:54:55,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:55,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362495862, max timestamp=1682362495867Num uniques 1018; 2023-04-24 18:54:55,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:55,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362495862, max timestamp=1682362495867Num uniques 1018; 2023-04-24 18:54:55,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:55,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362495880, max timestamp=1682362495887Num uniques 1027; 2023-04-24 18:54:55,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-24 18:54:55,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362495880, max timestamp=1682362495887Num uniques 1027; 2023-04-24 18:54:55,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-24 18:54:55,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682362495880, max timestamp=1682362495887Num uniques 1027; 2023-04-24 18:54:55,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-24 18:54:55,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362495894, max timestamp=1682362495907Num uniques 1136; 2023-04-24 18:54:55,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 18:54:55,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362495894, max timestamp=1682362495907Num uniques 1136; 2023-04-24 18:54:55,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362495894, max timestamp=1682362495907Num uniques 1136; 2023-04-24 18:54:55,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 18:54:55,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362495915, max timestamp=1682362495925Num uniques 1095; 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362495915, max timestamp=1682362495925Num uniques 1095; 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362495915, max timestamp=1682362495925Num uniques 1095; 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:55,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362495937, max timestamp=1682362495949Num uniques 1104; 2023-04-24 18:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 18:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362495937, max timestamp=1682362495949Num uniques 1104; 2023-04-24 18:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682362495937, max timestamp=1682362495949Num uniques 1104; 2023-04-24 18:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 18:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 18:54:56,009 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/22b206b25c2b4902bef97b47db063cfd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/22b206b25c2b4902bef97b47db063cfd 2023-04-24 18:54:56,016 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into 22b206b25c2b4902bef97b47db063cfd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:56,016 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:56,016 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=13, startTime=1682362495542; duration=0sec 2023-04-24 18:54:56,016 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1682362495988, max timestamp=1682362496001Num uniques 1013; 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1682362495988, max timestamp=1682362496001Num uniques 1013; 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362495988, max timestamp=1682362495993Num uniques 1013; 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:56,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 18:54:56,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682362496052, max timestamp=1682362496058Num uniques 1022; 2023-04-24 18:54:56,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-24 18:54:56,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682362496052, max timestamp=1682362496058Num uniques 1022; 2023-04-24 18:54:56,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682362496052, max timestamp=1682362496058Num uniques 1022; 2023-04-24 18:54:56,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 18:54:56,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-24 18:54:56,068 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/17297a17a44e496ea825fa8093dec2f0 2023-04-24 18:54:56,080 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/f96538941dd34094ae7cdc155c268242 2023-04-24 18:54:56,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362496071, max timestamp=1682362496079Num uniques 1031; 2023-04-24 18:54:56,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 18:54:56,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362496071, max timestamp=1682362496079Num uniques 1081; 2023-04-24 18:54:56,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682362496071, max timestamp=1682362496079Num uniques 1081; 2023-04-24 18:54:56,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-24 18:54:56,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-24 18:54:56,086 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/43d6d0b11174466d9b2a87cb7ae27835 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835 2023-04-24 18:54:56,090 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835, entries=100, sequenceid=534, filesize=9.5 K 2023-04-24 18:54:56,091 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/17297a17a44e496ea825fa8093dec2f0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0 2023-04-24 18:54:56,094 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0, entries=100, sequenceid=534, filesize=9.5 K 2023-04-24 18:54:56,095 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/f96538941dd34094ae7cdc155c268242 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242 2023-04-24 18:54:56,100 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242, entries=100, sequenceid=534, filesize=9.5 K 2023-04-24 18:54:56,101 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=140.89 KB/144270 for edaa58be4d671676fa0749811078529f in 452ms, sequenceid=534, compaction requested=false 2023-04-24 18:54:56,101 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:56,101 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:56,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=147.60 KB heapSize=418.78 KB 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:56,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 18:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,110 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476c48a8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362556114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,118 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,118 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362556115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362556116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,120 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,120 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,120 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,120 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,125 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,125 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:56,125 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,125 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,128 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362556218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362556220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362556220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,225 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,228 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,231 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,232 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362556230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362556232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,234 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,234 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,234 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705c08b3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,237 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,237 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,243 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,243 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,249 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,249 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,349 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,353 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,355 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,357 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2acd59ea[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,363 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,367 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,367 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,377 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,377 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:56,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,381 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,381 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362556421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362556424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362556424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,481 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,485 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,486 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,489 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,489 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbe089e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,495 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,498 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,498 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:56,504 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,505 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/eb0ca24931624b16bdccf0a6fb4c3e23 2023-04-24 18:54:56,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/0bf7597b2e244e98816ed170addba5d6 2023-04-24 18:54:56,605 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,608 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,612 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,612 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,615 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b238a9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,619 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,622 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,622 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,629 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,629 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362556725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362556727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:56,729 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362556728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,732 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,732 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,732 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,732 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,734 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,736 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,736 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,738 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39bba4bc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,746 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,746 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,755 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,755 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,755 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,755 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,756 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,756 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,856 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,859 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,861 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:56,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:56,864 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f9b868[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:56,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:56,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,867 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:56,868 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,868 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:56,870 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:56,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:56,873 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:56,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:56,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:56,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:56,874 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:56,875 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,875 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,875 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:56,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:56,876 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:56,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:56,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,880 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:56,880 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:56,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:56,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/8eeac9c21d6449889f7385c485c1894c 2023-04-24 18:54:56,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/eb0ca24931624b16bdccf0a6fb4c3e23 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23 2023-04-24 18:54:56,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23, entries=250, sequenceid=710, filesize=16.7 K 2023-04-24 18:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/0bf7597b2e244e98816ed170addba5d6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6 2023-04-24 18:54:56,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6, entries=150, sequenceid=710, filesize=11.9 K 2023-04-24 18:54:56,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/8eeac9c21d6449889f7385c485c1894c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c 2023-04-24 18:54:56,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c, entries=150, sequenceid=710, filesize=11.9 K 2023-04-24 18:54:56,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~418.73 KB/428784, currentSize=33.54 KB/34350 for edaa58be4d671676fa0749811078529f in 874ms, sequenceid=710, compaction requested=true 2023-04-24 18:54:56,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:56,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:56,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:56,978 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:56,978 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:56,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:56,978 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:56,979 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:56,979 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/96800baeb6ce40dd96f8bd110f0997f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=38.7 K 2023-04-24 18:54:56,979 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,979 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/22b206b25c2b4902bef97b47db063cfd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=34.0 K 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 96800baeb6ce40dd96f8bd110f0997f7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494077 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 22b206b25c2b4902bef97b47db063cfd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494077 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 43d6d0b11174466d9b2a87cb7ae27835, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682362494696 2023-04-24 18:54:56,979 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 17297a17a44e496ea825fa8093dec2f0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682362494696 2023-04-24 18:54:56,980 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting eb0ca24931624b16bdccf0a6fb4c3e23, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496094 2023-04-24 18:54:56,980 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0bf7597b2e244e98816ed170addba5d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496099 2023-04-24 18:54:56,981 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:56,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:56,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:56,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:56,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:56,984 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:56,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:56,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:56,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,987 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:56,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:56,987 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:56,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:56,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:56,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:56,988 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:56,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:57,001 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:57,003 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:57,022 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/11bb3b04fd5a4cdd9e243df2716af721 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11bb3b04fd5a4cdd9e243df2716af721 2023-04-24 18:54:57,025 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/7f39ac28a6ab486c8786eee410643752 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/7f39ac28a6ab486c8786eee410643752 2023-04-24 18:54:57,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into 11bb3b04fd5a4cdd9e243df2716af721(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:57,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:57,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=13, startTime=1682362496978; duration=0sec 2023-04-24 18:54:57,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:57,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:54:57,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:54:57,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:57,030 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:57,030 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c3b8d3265de8467fbbef058e4a8e0d34, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=34.0 K 2023-04-24 18:54:57,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c3b8d3265de8467fbbef058e4a8e0d34, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682362494077 2023-04-24 18:54:57,031 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f96538941dd34094ae7cdc155c268242, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682362494696 2023-04-24 18:54:57,031 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8eeac9c21d6449889f7385c485c1894c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496099 2023-04-24 18:54:57,033 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 7f39ac28a6ab486c8786eee410643752(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:57,033 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:57,033 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=13, startTime=1682362496978; duration=0sec 2023-04-24 18:54:57,033 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:57,042 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:57,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362497239, max timestamp=1682362497244Num uniques 859; 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362497239, max timestamp=1682362497244Num uniques 859; 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362497239, max timestamp=1682362497244Num uniques 859; 2023-04-24 18:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:57,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362497261, max timestamp=1682362497264Num uniques 1018; 2023-04-24 18:54:57,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:57,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362497261, max timestamp=1682362497264Num uniques 1018; 2023-04-24 18:54:57,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:57,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362497261, max timestamp=1682362497264Num uniques 1018; 2023-04-24 18:54:57,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:54:57,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362497276, max timestamp=1682362497283Num uniques 1027; 2023-04-24 18:54:57,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362497276, max timestamp=1682362497283Num uniques 1027; 2023-04-24 18:54:57,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:54:57,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362497276, max timestamp=1682362497283Num uniques 1027; 2023-04-24 18:54:57,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:54:57,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362497301, max timestamp=1682362497307Num uniques 1086; 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362497301, max timestamp=1682362497307Num uniques 1086; 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362497301, max timestamp=1682362497307Num uniques 1086; 2023-04-24 18:54:57,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 18:54:57,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362557331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362497326, max timestamp=1682362497333Num uniques 1095; 2023-04-24 18:54:57,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:57,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362497326, max timestamp=1682362497333Num uniques 1095; 2023-04-24 18:54:57,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:57,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682362497326, max timestamp=1682362497333Num uniques 1095; 2023-04-24 18:54:57,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 18:54:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362557367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362557368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,420 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/11c9044e5a664d11a3556f49749668df 2023-04-24 18:54:57,432 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/1d15f0a724af44c8816f58b16e599076 2023-04-24 18:54:57,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362557442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,464 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/b22e07c103894491819b6c01336242ff 2023-04-24 18:54:57,468 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/3d52b6f975de45fca0a43a0671fefee8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3d52b6f975de45fca0a43a0671fefee8 2023-04-24 18:54:57,471 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/11c9044e5a664d11a3556f49749668df as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df 2023-04-24 18:54:57,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362557472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362557473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,474 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into 3d52b6f975de45fca0a43a0671fefee8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:57,474 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:57,474 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=13, startTime=1682362496978; duration=0sec 2023-04-24 18:54:57,474 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:57,480 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df, entries=100, sequenceid=718, filesize=9.5 K 2023-04-24 18:54:57,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/1d15f0a724af44c8816f58b16e599076 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076 2023-04-24 18:54:57,486 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076, entries=100, sequenceid=718, filesize=9.5 K 2023-04-24 18:54:57,487 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/b22e07c103894491819b6c01336242ff as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff 2023-04-24 18:54:57,491 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff, entries=100, sequenceid=718, filesize=9.5 K 2023-04-24 18:54:57,492 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for edaa58be4d671676fa0749811078529f in 504ms, sequenceid=718, compaction requested=false 2023-04-24 18:54:57,492 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:57,492 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57fa531[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:57,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:57,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:57,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:57,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:57,502 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:57,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:57,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:57,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:57,503 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:57,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:57,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:57,505 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:57,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:57,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:57,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:57,511 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,511 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,511 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:57,511 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:57,511 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:57,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:57,511 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:57,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:57,511 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:57,511 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:57,611 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:57,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:57,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:57,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:57,615 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:57,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:57,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:57,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:57,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:57,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:57,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:57,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:57,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:57,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:57,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:57,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:57,618 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=147.60 KB heapSize=416.77 KB 2023-04-24 18:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:57,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:57,619 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:57,647 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:57,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362557660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362557675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362557675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362557762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362557964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362557978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:57,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362557979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,026 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0e00d340f2e942918abd7c7576e322ea 2023-04-24 18:54:58,036 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/db43e327dd244c3e8b5ce21cfac460c3 2023-04-24 18:54:58,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41346 deadline: 1682362558242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,245 DEBUG [Thread-1658] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:58,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41358 deadline: 1682362558244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,248 DEBUG [Thread-1656] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:54:58,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362558267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,447 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/bdc42184c24340a8b48dac0d8845ff50 2023-04-24 18:54:58,452 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0e00d340f2e942918abd7c7576e322ea as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea 2023-04-24 18:54:58,455 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea, entries=150, sequenceid=827, filesize=11.9 K 2023-04-24 18:54:58,456 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/db43e327dd244c3e8b5ce21cfac460c3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3 2023-04-24 18:54:58,460 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3, entries=150, sequenceid=827, filesize=11.9 K 2023-04-24 18:54:58,461 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/bdc42184c24340a8b48dac0d8845ff50 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50 2023-04-24 18:54:58,465 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50, entries=150, sequenceid=827, filesize=11.9 K 2023-04-24 18:54:58,465 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~416.72 KB/426720, currentSize=33.54 KB/34350 for edaa58be4d671676fa0749811078529f in 847ms, sequenceid=827, compaction requested=true 2023-04-24 18:54:58,466 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:58,466 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:58,467 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d30b247[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:58,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:58,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,470 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:58,471 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,471 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:58,476 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:58,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:58,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:58,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:58,476 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:58,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:58,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:58,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:58,482 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,483 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:58,483 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,482 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:58,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:58,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:58,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/9f0cfaf3184c46fabad0669b0109a918 2023-04-24 18:54:58,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/19c778de9c034750b616f5613f7d66f0 2023-04-24 18:54:58,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362558515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362498508, max timestamp=1682362498517Num uniques 859; 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362498508, max timestamp=1682362498517Num uniques 859; 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362498508, max timestamp=1682362498517Num uniques 859; 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:58,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:54:58,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362558574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,583 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:58,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:58,586 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:58,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:58,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:58,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,591 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,591 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,591 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:58,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:58,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:58,594 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ca7f19[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:58,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:58,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,597 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:58,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:58,598 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:58,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:58,601 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:58,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:58,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:58,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:58,601 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:58,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,605 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:58,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,608 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:58,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,608 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:58,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362558619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362558679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,708 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:58,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:58,711 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:58,711 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,711 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:58,711 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:58,712 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,712 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,713 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:58,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:58,715 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:58,715 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,716 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc4a6b0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:58,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:58,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,720 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:58,720 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,720 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:58,722 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,722 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,722 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:58,724 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:58,724 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:58,724 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:58,724 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:58,724 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:58,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,726 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,729 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,729 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,729 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,729 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:58,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:58,729 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,730 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:58,730 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362558770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362558823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,830 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:58,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:58,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:58,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:58,836 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:58,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:58,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:58,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:58,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:58,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:58,840 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,840 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:58,841 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:58,841 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,841 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:58,841 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:58,841 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,841 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:58,841 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:58,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,843 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15eac014[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:58,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:58,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:58,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:58,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:58,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,847 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:58,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:58,850 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:58,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:58,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:58,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:58,850 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:58,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,853 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,854 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:58,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:58,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:58,856 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:58,857 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:58,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:58,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:58,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362558883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/dfcb7a13d67b43199c9203fc792180f1 2023-04-24 18:54:58,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/9f0cfaf3184c46fabad0669b0109a918 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918 2023-04-24 18:54:58,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918, entries=150, sequenceid=838, filesize=11.9 K 2023-04-24 18:54:58,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/19c778de9c034750b616f5613f7d66f0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0 2023-04-24 18:54:58,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0, entries=150, sequenceid=838, filesize=11.9 K 2023-04-24 18:54:58,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/dfcb7a13d67b43199c9203fc792180f1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1 2023-04-24 18:54:58,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1, entries=150, sequenceid=838, filesize=11.9 K 2023-04-24 18:54:58,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for edaa58be4d671676fa0749811078529f in 441ms, sequenceid=838, compaction requested=true 2023-04-24 18:54:58,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:58,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:58,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:58,927 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:58,927 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:58,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:58,928 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:58,929 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:58,929 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:54:58,929 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:54:58,929 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,929 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/7f39ac28a6ab486c8786eee410643752, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=46.0 K 2023-04-24 18:54:58,929 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,929 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11bb3b04fd5a4cdd9e243df2716af721, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=46.0 K 2023-04-24 18:54:58,929 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7f39ac28a6ab486c8786eee410643752, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496099 2023-04-24 18:54:58,930 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 11bb3b04fd5a4cdd9e243df2716af721, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496099 2023-04-24 18:54:58,930 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 11c9044e5a664d11a3556f49749668df, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682362496113 2023-04-24 18:54:58,930 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1d15f0a724af44c8816f58b16e599076, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682362496113 2023-04-24 18:54:58,930 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0e00d340f2e942918abd7c7576e322ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682362497363 2023-04-24 18:54:58,931 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting db43e327dd244c3e8b5ce21cfac460c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682362497363 2023-04-24 18:54:58,931 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9f0cfaf3184c46fabad0669b0109a918, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:54:58,931 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 19c778de9c034750b616f5613f7d66f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:54:58,938 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:58,938 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:58,950 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/50868ff354254bff975756945b1d06cd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50868ff354254bff975756945b1d06cd 2023-04-24 18:54:58,954 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 50868ff354254bff975756945b1d06cd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:58,954 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:58,954 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=12, startTime=1682362498927; duration=0sec 2023-04-24 18:54:58,954 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:54:58,954 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:54:58,956 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:54:58,956 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:54:58,956 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,956 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3d52b6f975de45fca0a43a0671fefee8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=46.0 K 2023-04-24 18:54:58,956 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3d52b6f975de45fca0a43a0671fefee8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1682362496099 2023-04-24 18:54:58,957 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:58,957 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b22e07c103894491819b6c01336242ff, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682362496113 2023-04-24 18:54:58,957 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting bdc42184c24340a8b48dac0d8845ff50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682362497363 2023-04-24 18:54:58,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:58,958 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting dfcb7a13d67b43199c9203fc792180f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:54:58,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:58,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:58,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:58,970 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:58,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:58,970 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:58,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,971 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:54:58,972 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,972 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:58,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:58,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:58,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:58,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:58,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:58,974 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:58,974 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:58,974 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:54:58,980 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/919c3b2c2f754db0ada6bffe98089dd1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/919c3b2c2f754db0ada6bffe98089dd1 2023-04-24 18:54:58,994 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into 919c3b2c2f754db0ada6bffe98089dd1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:58,994 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:58,994 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=12, startTime=1682362498927; duration=0sec 2023-04-24 18:54:58,994 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:54:59,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:54:59,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362559151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362559187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362559254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,351 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/0b81c79e47854bcfa5fd66598decadc8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0b81c79e47854bcfa5fd66598decadc8 2023-04-24 18:54:59,354 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into 0b81c79e47854bcfa5fd66598decadc8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:54:59,355 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:59,355 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=12, startTime=1682362498927; duration=0sec 2023-04-24 18:54:59,355 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:54:59,395 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0a55a5e7f571494b816fd7c3580805b3 2023-04-24 18:54:59,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362559456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362559692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362559758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:54:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362559777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,805 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/c722631c46a8408b9a7c9b031b5b3973 2023-04-24 18:54:59,816 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/81b4cdb439d043208283b6d025d58cee 2023-04-24 18:54:59,819 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/0a55a5e7f571494b816fd7c3580805b3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3 2023-04-24 18:54:59,823 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3, entries=150, sequenceid=875, filesize=11.9 K 2023-04-24 18:54:59,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/c722631c46a8408b9a7c9b031b5b3973 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973 2023-04-24 18:54:59,827 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973, entries=150, sequenceid=875, filesize=11.9 K 2023-04-24 18:54:59,828 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/81b4cdb439d043208283b6d025d58cee as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee 2023-04-24 18:54:59,831 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee, entries=150, sequenceid=875, filesize=11.9 K 2023-04-24 18:54:59,832 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for edaa58be4d671676fa0749811078529f in 858ms, sequenceid=875, compaction requested=false 2023-04-24 18:54:59,832 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:54:59,832 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:54:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:54:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:54:59,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:54:59,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:59,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:59,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:54:59,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d70fa64[Count = 0] remaining members to acquire global barrier 2023-04-24 18:54:59,838 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:54:59,838 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:54:59,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:54:59,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:54:59,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:54:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:54:59,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:54:59,845 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:54:59,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:54:59,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:54:59,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:54:59,845 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:54:59,846 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,846 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,846 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:54:59,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:54:59,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:59,847 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:54:59,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:54:59,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:59,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:59,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:54:59,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:59,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:59,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:54:59,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:54:59,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:54:59,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:54:59,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,851 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:54:59,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:54:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:54:59,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:54:59,851 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:54:59,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,951 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:54:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:54:59,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:54:59,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:54:59,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:54:59,955 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:54:59,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:59,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:54:59,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:54:59,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:59,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:54:59,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:54:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:54:59,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:54:59,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:59,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:54:59,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:54:59,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:54:59,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:54:59,960 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:54:59,960 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:00,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:55:00,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362500289, max timestamp=1682362500305Num uniques 859; 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362500289, max timestamp=1682362500305Num uniques 859; 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362500289, max timestamp=1682362500305Num uniques 859; 2023-04-24 18:55:00,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:00,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362560348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:00,394 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/e0bc9005d81f44fea8cd31922dbc1db9 2023-04-24 18:55:00,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:00,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362560451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:00,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362560653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:00,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41374 deadline: 1682362560701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:00,805 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/60bdbbeb92dc4057aabf33abda4c52b1 2023-04-24 18:55:00,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:00,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362560956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,225 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/3029e15b48e14bb7aea51f375c42f4d9 2023-04-24 18:55:01,229 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/e0bc9005d81f44fea8cd31922dbc1db9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9 2023-04-24 18:55:01,232 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9, entries=100, sequenceid=888, filesize=9.5 K 2023-04-24 18:55:01,233 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/60bdbbeb92dc4057aabf33abda4c52b1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1 2023-04-24 18:55:01,240 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1, entries=100, sequenceid=888, filesize=9.5 K 2023-04-24 18:55:01,241 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/3029e15b48e14bb7aea51f375c42f4d9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9 2023-04-24 18:55:01,245 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9, entries=100, sequenceid=888, filesize=9.5 K 2023-04-24 18:55:01,245 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for edaa58be4d671676fa0749811078529f in 1286ms, sequenceid=888, compaction requested=true 2023-04-24 18:55:01,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:01,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:01,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c86a64[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:01,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:01,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,251 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:01,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,252 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:01,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:01,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:01,255 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:01,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:01,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:01,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:01,256 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:01,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:01,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:01,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,259 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:01,267 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:01,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:01,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:01,267 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:01,268 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:01,268 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:01,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:01,368 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:01,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:01,371 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:01,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:01,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:01,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:01,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:01,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:01,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:01,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:01,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:01,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:01,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:01,374 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. started... 2023-04-24 18:55:01,375 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:01,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:55:01,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. as already flushing 2023-04-24 18:55:01,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:01,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362561481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:01,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362561585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,785 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=924 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/b1cb18e541474d22b0287b17c50efad3 2023-04-24 18:55:01,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:01,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41420 deadline: 1682362561784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,787 DEBUG [Thread-1654] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:01,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362561789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:01,802 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=924 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ca78331e0fac42aa8bfc9855a96aa747 2023-04-24 18:55:01,808 DEBUG [Thread-1669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:59583 2023-04-24 18:55:01,809 DEBUG [Thread-1669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:01,810 DEBUG [Thread-1667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:59583 2023-04-24 18:55:01,810 DEBUG [Thread-1667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:01,811 DEBUG [Thread-1671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:59583 2023-04-24 18:55:01,811 DEBUG [Thread-1671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:01,813 DEBUG [Thread-1665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:59583 2023-04-24 18:55:01,813 DEBUG [Thread-1673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:59583 2023-04-24 18:55:01,813 DEBUG [Thread-1665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:01,813 DEBUG [Thread-1673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:02,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:02,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41434 deadline: 1682362562092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,216 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=924 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/3a65208cfc1c4c80a2b9eb9c65f1b86a 2023-04-24 18:55:02,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/b1cb18e541474d22b0287b17c50efad3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3 2023-04-24 18:55:02,231 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3, entries=150, sequenceid=924, filesize=11.9 K 2023-04-24 18:55:02,232 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/ca78331e0fac42aa8bfc9855a96aa747 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747 2023-04-24 18:55:02,236 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747, entries=150, sequenceid=924, filesize=11.9 K 2023-04-24 18:55:02,237 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/3a65208cfc1c4c80a2b9eb9c65f1b86a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a 2023-04-24 18:55:02,241 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a, entries=150, sequenceid=924, filesize=11.9 K 2023-04-24 18:55:02,242 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for edaa58be4d671676fa0749811078529f in 867ms, sequenceid=924, compaction requested=true 2023-04-24 18:55:02,242 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:02,242 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:02,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:02,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:02,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:02,242 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff1b90b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:02,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:02,246 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:02,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:02,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,248 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:02,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:02,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:02,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:02,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:02,251 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:02,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:02,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:02,251 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:02,251 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:02,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:02,253 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:02,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:02,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,255 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,256 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:02,256 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:02,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:02,256 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:02,256 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:02,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:02,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:02,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:02,257 DEBUG [Thread-1664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:02,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:02,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:02,257 DEBUG [Thread-1664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:02,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on edaa58be4d671676fa0749811078529f 2023-04-24 18:55:02,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:02,271 DEBUG [Thread-1656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:59583 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:55:02,271 DEBUG [Thread-1656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:55:02,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:02,285 DEBUG [Thread-1658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:59583 2023-04-24 18:55:02,285 DEBUG [Thread-1658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:02,357 DEBUG [Thread-1664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:02,599 DEBUG [Thread-1660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:59583 2023-04-24 18:55:02,599 DEBUG [Thread-1660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:02,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/50e170d2ff244602821035b4cec18206 2023-04-24 18:55:02,721 DEBUG [Thread-1662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:59583 2023-04-24 18:55:02,721 DEBUG [Thread-1662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:03,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/f833747ba9bd46d3b4b83ea53ab4473b 2023-04-24 18:55:03,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/105120edfa9f4eb0a20d978c39f0f79f 2023-04-24 18:55:03,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/50e170d2ff244602821035b4cec18206 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206 2023-04-24 18:55:03,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206, entries=150, sequenceid=936, filesize=11.9 K 2023-04-24 18:55:03,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/f833747ba9bd46d3b4b83ea53ab4473b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b 2023-04-24 18:55:03,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b, entries=150, sequenceid=936, filesize=11.9 K 2023-04-24 18:55:03,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/105120edfa9f4eb0a20d978c39f0f79f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f 2023-04-24 18:55:03,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f, entries=150, sequenceid=936, filesize=11.9 K 2023-04-24 18:55:03,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for edaa58be4d671676fa0749811078529f in 1240ms, sequenceid=936, compaction requested=true 2023-04-24 18:55:03,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:03,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:03,511 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:03,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:03,511 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:03,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59406 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/A is initiating minor compaction (all files) 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59406 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:03,513 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/A in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/B is initiating minor compaction (all files) 2023-04-24 18:55:03,513 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50868ff354254bff975756945b1d06cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=58.0 K 2023-04-24 18:55:03,513 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/B in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:03,513 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0b81c79e47854bcfa5fd66598decadc8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=58.0 K 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 50868ff354254bff975756945b1d06cd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:55:03,513 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0b81c79e47854bcfa5fd66598decadc8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:55:03,514 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0a55a5e7f571494b816fd7c3580805b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1682362498553 2023-04-24 18:55:03,514 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c722631c46a8408b9a7c9b031b5b3973, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1682362498553 2023-04-24 18:55:03,514 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e0bc9005d81f44fea8cd31922dbc1db9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682362499148 2023-04-24 18:55:03,514 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 60bdbbeb92dc4057aabf33abda4c52b1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682362499148 2023-04-24 18:55:03,514 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b1cb18e541474d22b0287b17c50efad3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=924, earliestPutTs=1682362500331 2023-04-24 18:55:03,515 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ca78331e0fac42aa8bfc9855a96aa747, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=924, earliestPutTs=1682362500331 2023-04-24 18:55:03,515 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 50e170d2ff244602821035b4cec18206, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682362501477 2023-04-24 18:55:03,515 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f833747ba9bd46d3b4b83ea53ab4473b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682362501477 2023-04-24 18:55:03,529 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#B#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:03,531 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#A#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:03,538 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/86a93db8f585472aa40bd19473102e6d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/86a93db8f585472aa40bd19473102e6d 2023-04-24 18:55:03,542 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in edaa58be4d671676fa0749811078529f/A of edaa58be4d671676fa0749811078529f into 86a93db8f585472aa40bd19473102e6d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:03,542 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:03,542 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/A, priority=11, startTime=1682362503511; duration=0sec 2023-04-24 18:55:03,542 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:03,542 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:03,543 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59406 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:03,543 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): edaa58be4d671676fa0749811078529f/C is initiating minor compaction (all files) 2023-04-24 18:55:03,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of edaa58be4d671676fa0749811078529f/C in TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:03,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/919c3b2c2f754db0ada6bffe98089dd1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp, totalSize=58.0 K 2023-04-24 18:55:03,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 919c3b2c2f754db0ada6bffe98089dd1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682362497659 2023-04-24 18:55:03,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 81b4cdb439d043208283b6d025d58cee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1682362498553 2023-04-24 18:55:03,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3029e15b48e14bb7aea51f375c42f4d9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682362499148 2023-04-24 18:55:03,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3a65208cfc1c4c80a2b9eb9c65f1b86a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=924, earliestPutTs=1682362500331 2023-04-24 18:55:03,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 105120edfa9f4eb0a20d978c39f0f79f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1682362501477 2023-04-24 18:55:03,553 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): edaa58be4d671676fa0749811078529f#C#compaction#302 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:03,937 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/d80a2fa3b97249cf84566ecaf7cb01c6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d80a2fa3b97249cf84566ecaf7cb01c6 2023-04-24 18:55:03,941 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in edaa58be4d671676fa0749811078529f/B of edaa58be4d671676fa0749811078529f into d80a2fa3b97249cf84566ecaf7cb01c6(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:03,942 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:03,942 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/B, priority=11, startTime=1682362503511; duration=0sec 2023-04-24 18:55:03,942 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:03,961 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/55b9453e751f47fa8fd4c29aec345ae9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/55b9453e751f47fa8fd4c29aec345ae9 2023-04-24 18:55:03,965 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in edaa58be4d671676fa0749811078529f/C of edaa58be4d671676fa0749811078529f into 55b9453e751f47fa8fd4c29aec345ae9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:03,965 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:03,965 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f., storeName=edaa58be4d671676fa0749811078529f/C, priority=11, startTime=1682362503511; duration=0sec 2023-04-24 18:55:03,965 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:05,821 DEBUG [Thread-1654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:59583 2023-04-24 18:55:05,821 DEBUG [Thread-1654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 190 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 185 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:55:05,821 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5439 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5323 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5038 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5441 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 5315 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:55:05,822 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:59583 2023-04-24 18:55:05,822 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:05,822 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:55:05,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:55:05,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:05,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:05,826 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362505826"}]},"ts":"1682362505826"} 2023-04-24 18:55:05,827 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:55:05,832 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:55:05,833 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, UNASSIGN}] 2023-04-24 18:55:05,834 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, UNASSIGN 2023-04-24 18:55:05,835 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=edaa58be4d671676fa0749811078529f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:05,835 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362505835"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362505835"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362505835"}]},"ts":"1682362505835"} 2023-04-24 18:55:05,836 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:05,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:05,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close edaa58be4d671676fa0749811078529f 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing edaa58be4d671676fa0749811078529f, disabling compactions & flushes 2023-04-24 18:55:05,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. after waiting 0 ms 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:05,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing edaa58be4d671676fa0749811078529f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=A 2023-04-24 18:55:05,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:05,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=B 2023-04-24 18:55:05,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:05,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK edaa58be4d671676fa0749811078529f, store=C 2023-04-24 18:55:05,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:06,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:06,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=946 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2b058b9a9d06404e858a2bdb87edb7ea 2023-04-24 18:55:06,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:06,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=946 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/d81c54a694c64fb68138a18f7ed87f42 2023-04-24 18:55:06,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:07,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=946 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/c1b97def993c4696a5756a5bd3a3f4e3 2023-04-24 18:55:07,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/A/2b058b9a9d06404e858a2bdb87edb7ea as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2b058b9a9d06404e858a2bdb87edb7ea 2023-04-24 18:55:07,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2b058b9a9d06404e858a2bdb87edb7ea, entries=150, sequenceid=946, filesize=11.9 K 2023-04-24 18:55:07,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/B/d81c54a694c64fb68138a18f7ed87f42 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d81c54a694c64fb68138a18f7ed87f42 2023-04-24 18:55:07,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d81c54a694c64fb68138a18f7ed87f42, entries=150, sequenceid=946, filesize=11.9 K 2023-04-24 18:55:07,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/.tmp/C/c1b97def993c4696a5756a5bd3a3f4e3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c1b97def993c4696a5756a5bd3a3f4e3 2023-04-24 18:55:07,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c1b97def993c4696a5756a5bd3a3f4e3, entries=150, sequenceid=946, filesize=11.9 K 2023-04-24 18:55:07,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for edaa58be4d671676fa0749811078529f in 1239ms, sequenceid=946, compaction requested=false 2023-04-24 18:55:07,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9196e81d175f4b778c77ce46ac8983ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/652dcc2825f84b69a2d6c168c14bc38f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/896194be52fc4672b333f64d49bf674d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/96800baeb6ce40dd96f8bd110f0997f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/7f39ac28a6ab486c8786eee410643752, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50868ff354254bff975756945b1d06cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206] to archive 2023-04-24 18:55:07,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:07,251 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cc3dbd55e8d94c08af9700fa5c56685e 2023-04-24 18:55:07,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/5702ef2b1d69483da1d06733e40c0c40 2023-04-24 18:55:07,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/6b7a1e3db10d4020aed5d7de70470a49 2023-04-24 18:55:07,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/d80cb0c8f76c4bf3ba2e045f1eebdc56 2023-04-24 18:55:07,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9196e81d175f4b778c77ce46ac8983ea to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9196e81d175f4b778c77ce46ac8983ea 2023-04-24 18:55:07,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9cf55328d6af4d7bb98c99984d3f01b8 2023-04-24 18:55:07,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1ae100ecfe9846b5b6eaf3cec378c4a6 2023-04-24 18:55:07,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/203a805f187d4e859290e42b940de087 2023-04-24 18:55:07,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/8f3de500995f47a9aba99250f0b76508 2023-04-24 18:55:07,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/92047f3618414f3eaf38e2977558566c 2023-04-24 18:55:07,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/cfcb20933aa5415a99c6d94db511ff14 2023-04-24 18:55:07,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2ee2f6a732a142d898dc33d0a78383ce 2023-04-24 18:55:07,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a931a907f6a47679f704ff83f3b703c 2023-04-24 18:55:07,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/1b53b298d98f41e8b7c668fe9934f038 2023-04-24 18:55:07,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/652dcc2825f84b69a2d6c168c14bc38f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/652dcc2825f84b69a2d6c168c14bc38f 2023-04-24 18:55:07,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/3bccc67c8cba4f7d9c7d6279e130b30a 2023-04-24 18:55:07,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/896194be52fc4672b333f64d49bf674d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/896194be52fc4672b333f64d49bf674d 2023-04-24 18:55:07,265 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b560a9fc02f34b3592c36a93bbbeacdc 2023-04-24 18:55:07,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/f799a252b674401ba23ad5261c678d96 2023-04-24 18:55:07,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2969f0db926144ce969ab514b2dee43d 2023-04-24 18:55:07,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/96800baeb6ce40dd96f8bd110f0997f7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/96800baeb6ce40dd96f8bd110f0997f7 2023-04-24 18:55:07,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/43d6d0b11174466d9b2a87cb7ae27835 2023-04-24 18:55:07,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/eb0ca24931624b16bdccf0a6fb4c3e23 2023-04-24 18:55:07,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/7f39ac28a6ab486c8786eee410643752 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/7f39ac28a6ab486c8786eee410643752 2023-04-24 18:55:07,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/11c9044e5a664d11a3556f49749668df 2023-04-24 18:55:07,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0e00d340f2e942918abd7c7576e322ea 2023-04-24 18:55:07,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50868ff354254bff975756945b1d06cd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50868ff354254bff975756945b1d06cd 2023-04-24 18:55:07,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/9f0cfaf3184c46fabad0669b0109a918 2023-04-24 18:55:07,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/0a55a5e7f571494b816fd7c3580805b3 2023-04-24 18:55:07,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/e0bc9005d81f44fea8cd31922dbc1db9 2023-04-24 18:55:07,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/b1cb18e541474d22b0287b17c50efad3 2023-04-24 18:55:07,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/50e170d2ff244602821035b4cec18206 2023-04-24 18:55:07,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/39c5867437664940ad863d3256e8ef89, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/a6d7add99d9242daac5e09661e708c48, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ada1cd50099040aba493b871ee6596e1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/22b206b25c2b4902bef97b47db063cfd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11bb3b04fd5a4cdd9e243df2716af721, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0b81c79e47854bcfa5fd66598decadc8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b] to archive 2023-04-24 18:55:07,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:07,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/90ce0729862e4b20a0d6a5a6de00e08a 2023-04-24 18:55:07,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/334b0511bffe4bb8af05a3a5baee7da5 2023-04-24 18:55:07,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/43abe2712c374adaa23cf0895ba57f06 2023-04-24 18:55:07,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/39c5867437664940ad863d3256e8ef89 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/39c5867437664940ad863d3256e8ef89 2023-04-24 18:55:07,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/b61e569e47be4a16ac22183f19c58155 2023-04-24 18:55:07,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/e0ab370c107b4a7e99306f33881a3bae 2023-04-24 18:55:07,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11b35911f79d4ec39ff7c85e55bbf466 2023-04-24 18:55:07,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/fe0722e88d5440f3b2a06d83f172f8f7 2023-04-24 18:55:07,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/23c77bbd96994d219d4039f6e324de7b 2023-04-24 18:55:07,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/98901882e417452c9d02757982652bca 2023-04-24 18:55:07,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0e9908959fa44b218684e3d9396a3b14 2023-04-24 18:55:07,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/636fde17bc484cdf89bb865b949d41a4 2023-04-24 18:55:07,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/78ac6fab66bb429886e2f490f24fd21a 2023-04-24 18:55:07,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/a6d7add99d9242daac5e09661e708c48 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/a6d7add99d9242daac5e09661e708c48 2023-04-24 18:55:07,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ead69b87e949454db4975eb519cc867a 2023-04-24 18:55:07,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/87728e5103794cd887cf4d6a6445a62a 2023-04-24 18:55:07,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ada1cd50099040aba493b871ee6596e1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ada1cd50099040aba493b871ee6596e1 2023-04-24 18:55:07,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ba548269aea141f19e04c289de26a8df 2023-04-24 18:55:07,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/3fac3761626a4d7c8f56ad38dbaf4b16 2023-04-24 18:55:07,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/22b206b25c2b4902bef97b47db063cfd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/22b206b25c2b4902bef97b47db063cfd 2023-04-24 18:55:07,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f6c75577072e48e9abd3212d0677c809 2023-04-24 18:55:07,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/17297a17a44e496ea825fa8093dec2f0 2023-04-24 18:55:07,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11bb3b04fd5a4cdd9e243df2716af721 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/11bb3b04fd5a4cdd9e243df2716af721 2023-04-24 18:55:07,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0bf7597b2e244e98816ed170addba5d6 2023-04-24 18:55:07,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/1d15f0a724af44c8816f58b16e599076 2023-04-24 18:55:07,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/db43e327dd244c3e8b5ce21cfac460c3 2023-04-24 18:55:07,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0b81c79e47854bcfa5fd66598decadc8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/0b81c79e47854bcfa5fd66598decadc8 2023-04-24 18:55:07,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/19c778de9c034750b616f5613f7d66f0 2023-04-24 18:55:07,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/c722631c46a8408b9a7c9b031b5b3973 2023-04-24 18:55:07,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/60bdbbeb92dc4057aabf33abda4c52b1 2023-04-24 18:55:07,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/ca78331e0fac42aa8bfc9855a96aa747 2023-04-24 18:55:07,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/f833747ba9bd46d3b4b83ea53ab4473b 2023-04-24 18:55:07,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/d43afcc4267e49a2adc49d7d32b6f0c5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8cfe9bee1b7a4abc87599373197040bd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/020b8a22cc384accb3b737c6b175fa89, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c3b8d3265de8467fbbef058e4a8e0d34, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3d52b6f975de45fca0a43a0671fefee8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/919c3b2c2f754db0ada6bffe98089dd1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f] to archive 2023-04-24 18:55:07,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:07,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/782b50b280ac49fa91f6420ecbe5767a 2023-04-24 18:55:07,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/e038056170154796aa01517790226a6b 2023-04-24 18:55:07,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/ec9283c689e142b2b4a094c055e53e56 2023-04-24 18:55:07,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/d43afcc4267e49a2adc49d7d32b6f0c5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/d43afcc4267e49a2adc49d7d32b6f0c5 2023-04-24 18:55:07,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/da06b615782e47cfba820db35dd17c59 2023-04-24 18:55:07,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/296ba31471c24558b77bce758974bbb1 2023-04-24 18:55:07,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/632720fcf3fd4eb88662d54aa6e9ffcb 2023-04-24 18:55:07,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/6a55cf1a8b7348bfbcf6492e3a6b24ae 2023-04-24 18:55:07,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/7bf7777bc5bc491c9b1aa66b6cb1a009 2023-04-24 18:55:07,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8d02254218ed4ed6876ff453c5c7a653 2023-04-24 18:55:07,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/a5102e560be34f079869b366d336373b 2023-04-24 18:55:07,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/9cf11d9044214714849b1341e56a7d01 2023-04-24 18:55:07,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/fb37d69e6c3f4e6e8f99376c6b521d9b 2023-04-24 18:55:07,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8cfe9bee1b7a4abc87599373197040bd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8cfe9bee1b7a4abc87599373197040bd 2023-04-24 18:55:07,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b8ed4a12025d4eff96983879e1e88e9e 2023-04-24 18:55:07,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/43729b4994774ee492d50c03f273fcac 2023-04-24 18:55:07,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/020b8a22cc384accb3b737c6b175fa89 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/020b8a22cc384accb3b737c6b175fa89 2023-04-24 18:55:07,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5eb48f60a0cd47e4b08f61bdc159a8af 2023-04-24 18:55:07,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/5294c7fb795f44a89a19476f02a7d86d 2023-04-24 18:55:07,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c3b8d3265de8467fbbef058e4a8e0d34 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c3b8d3265de8467fbbef058e4a8e0d34 2023-04-24 18:55:07,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/506fd31df0e84ef996bc6f94a9e6e86a 2023-04-24 18:55:07,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/f96538941dd34094ae7cdc155c268242 2023-04-24 18:55:07,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3d52b6f975de45fca0a43a0671fefee8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3d52b6f975de45fca0a43a0671fefee8 2023-04-24 18:55:07,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/8eeac9c21d6449889f7385c485c1894c 2023-04-24 18:55:07,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/b22e07c103894491819b6c01336242ff 2023-04-24 18:55:07,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/bdc42184c24340a8b48dac0d8845ff50 2023-04-24 18:55:07,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/919c3b2c2f754db0ada6bffe98089dd1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/919c3b2c2f754db0ada6bffe98089dd1 2023-04-24 18:55:07,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/dfcb7a13d67b43199c9203fc792180f1 2023-04-24 18:55:07,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/81b4cdb439d043208283b6d025d58cee 2023-04-24 18:55:07,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3029e15b48e14bb7aea51f375c42f4d9 2023-04-24 18:55:07,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/3a65208cfc1c4c80a2b9eb9c65f1b86a 2023-04-24 18:55:07,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/105120edfa9f4eb0a20d978c39f0f79f 2023-04-24 18:55:07,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/recovered.edits/949.seqid, newMaxSeqId=949, maxSeqId=1 2023-04-24 18:55:07,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f. 2023-04-24 18:55:07,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for edaa58be4d671676fa0749811078529f: 2023-04-24 18:55:07,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed edaa58be4d671676fa0749811078529f 2023-04-24 18:55:07,348 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=edaa58be4d671676fa0749811078529f, regionState=CLOSED 2023-04-24 18:55:07,348 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362507348"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362507348"}]},"ts":"1682362507348"} 2023-04-24 18:55:07,350 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-24 18:55:07,350 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure edaa58be4d671676fa0749811078529f, server=jenkins-hbase4.apache.org,37593,1682362415680 in 1.5130 sec 2023-04-24 18:55:07,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-24 18:55:07,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=edaa58be4d671676fa0749811078529f, UNASSIGN in 1.5170 sec 2023-04-24 18:55:07,352 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362507352"}]},"ts":"1682362507352"} 2023-04-24 18:55:07,353 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:55:07,355 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:55:07,356 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5320 sec 2023-04-24 18:55:07,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 18:55:07,930 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-24 18:55:07,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:55:07,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,933 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-24 18:55:07,933 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,937 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:55:07,938 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/recovered.edits] 2023-04-24 18:55:07,941 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2b058b9a9d06404e858a2bdb87edb7ea to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/2b058b9a9d06404e858a2bdb87edb7ea 2023-04-24 18:55:07,942 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/86a93db8f585472aa40bd19473102e6d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/A/86a93db8f585472aa40bd19473102e6d 2023-04-24 18:55:07,944 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d80a2fa3b97249cf84566ecaf7cb01c6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d80a2fa3b97249cf84566ecaf7cb01c6 2023-04-24 18:55:07,945 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d81c54a694c64fb68138a18f7ed87f42 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/B/d81c54a694c64fb68138a18f7ed87f42 2023-04-24 18:55:07,947 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/55b9453e751f47fa8fd4c29aec345ae9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/55b9453e751f47fa8fd4c29aec345ae9 2023-04-24 18:55:07,949 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c1b97def993c4696a5756a5bd3a3f4e3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/C/c1b97def993c4696a5756a5bd3a3f4e3 2023-04-24 18:55:07,951 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/recovered.edits/949.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f/recovered.edits/949.seqid 2023-04-24 18:55:07,951 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/edaa58be4d671676fa0749811078529f 2023-04-24 18:55:07,951 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:55:07,954 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,955 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:55:07,956 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:55:07,957 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,957 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:55:07,958 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362507957"}]},"ts":"9223372036854775807"} 2023-04-24 18:55:07,959 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:55:07,959 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => edaa58be4d671676fa0749811078529f, NAME => 'TestAcidGuarantees,,1682362479641.edaa58be4d671676fa0749811078529f.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:55:07,959 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:55:07,959 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362507959"}]},"ts":"9223372036854775807"} 2023-04-24 18:55:07,960 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:55:07,962 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:07,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-04-24 18:55:08,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-24 18:55:08,034 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-24 18:55:08,045 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=558 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 378), ProcessCount=170 (was 170), AvailableMemoryMB=4221 (was 4258) 2023-04-24 18:55:08,056 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226, OpenFileDescriptor=558, MaxFileDescriptor=60000, SystemLoadAverage=371, ProcessCount=170, AvailableMemoryMB=4220 2023-04-24 18:55:08,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:55:08,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:55:08,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:08,060 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:55:08,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-24 18:55:08,061 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:55:08,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:08,062 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:08,062 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 empty. 2023-04-24 18:55:08,063 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:08,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:08,470 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:55:08,471 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7560c03701fe10683347e72e3a84b287, NAME => 'TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:55:08,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7560c03701fe10683347e72e3a84b287, disabling compactions & flushes 2023-04-24 18:55:08,877 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. after waiting 0 ms 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:08,877 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:08,877 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:08,879 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:55:08,880 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362508880"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362508880"}]},"ts":"1682362508880"} 2023-04-24 18:55:08,881 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:55:08,882 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:55:08,882 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362508882"}]},"ts":"1682362508882"} 2023-04-24 18:55:08,882 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:55:08,885 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, ASSIGN}] 2023-04-24 18:55:08,887 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, ASSIGN 2023-04-24 18:55:08,887 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:55:09,038 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:09,038 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362509038"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362509038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362509038"}]},"ts":"1682362509038"} 2023-04-24 18:55:09,040 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:09,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:09,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:09,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7560c03701fe10683347e72e3a84b287, NAME => 'TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:55:09,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:55:09,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,196 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,197 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A 2023-04-24 18:55:09,197 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A 2023-04-24 18:55:09,197 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:09,197 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName A 2023-04-24 18:55:09,197 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:09,198 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,198 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B 2023-04-24 18:55:09,198 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B 2023-04-24 18:55:09,199 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:09,199 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName B 2023-04-24 18:55:09,199 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:09,199 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,200 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C 2023-04-24 18:55:09,200 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C 2023-04-24 18:55:09,200 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:09,200 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName C 2023-04-24 18:55:09,201 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:09,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:09,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:55:09,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:09,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:55:09,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7560c03701fe10683347e72e3a84b287; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72265539, jitterRate=0.07684044539928436}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:55:09,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:09,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., pid=37, masterSystemTime=1682362509191 2023-04-24 18:55:09,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:09,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:09,208 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:09,208 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362509208"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362509208"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362509208"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362509208"}]},"ts":"1682362509208"} 2023-04-24 18:55:09,210 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-24 18:55:09,210 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 in 170 msec 2023-04-24 18:55:09,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-24 18:55:09,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, ASSIGN in 325 msec 2023-04-24 18:55:09,212 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:55:09,212 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362509212"}]},"ts":"1682362509212"} 2023-04-24 18:55:09,213 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:55:09,215 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:55:09,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-24 18:55:10,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 18:55:10,165 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-24 18:55:10,166 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:10,171 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:10,172 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:10,174 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:10,175 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:55:10,176 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:55:10,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:55:10,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:55:10,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:10,587 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:55:10,588 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:55:10,588 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:55:10,590 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 18:55:10,592 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, REOPEN/MOVE}] 2023-04-24 18:55:10,593 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, REOPEN/MOVE 2023-04-24 18:55:10,593 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:10,593 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362510593"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362510593"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362510593"}]},"ts":"1682362510593"} 2023-04-24 18:55:10,594 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:10,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:10,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7560c03701fe10683347e72e3a84b287, disabling compactions & flushes 2023-04-24 18:55:10,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:10,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:10,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. after waiting 0 ms 2023-04-24 18:55:10,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:10,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 18:55:10,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:10,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:10,750 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7560c03701fe10683347e72e3a84b287 to self. 2023-04-24 18:55:10,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:10,752 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=CLOSED 2023-04-24 18:55:10,752 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362510752"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362510752"}]},"ts":"1682362510752"} 2023-04-24 18:55:10,754 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-24 18:55:10,754 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 in 159 msec 2023-04-24 18:55:10,754 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=true 2023-04-24 18:55:10,904 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:10,905 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362510904"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362510904"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362510904"}]},"ts":"1682362510904"} 2023-04-24 18:55:10,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:11,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7560c03701fe10683347e72e3a84b287, NAME => 'TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:55:11,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:55:11,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,063 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,063 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A 2023-04-24 18:55:11,064 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A 2023-04-24 18:55:11,064 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:11,064 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName A 2023-04-24 18:55:11,065 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:11,066 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,066 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B 2023-04-24 18:55:11,066 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B 2023-04-24 18:55:11,066 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:11,067 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName B 2023-04-24 18:55:11,067 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:11,067 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,068 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C 2023-04-24 18:55:11,068 DEBUG [StoreOpener-7560c03701fe10683347e72e3a84b287-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C 2023-04-24 18:55:11,068 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:11,068 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7560c03701fe10683347e72e3a84b287 columnFamilyName C 2023-04-24 18:55:11,069 INFO [StoreOpener-7560c03701fe10683347e72e3a84b287-1] regionserver.HStore(310): Store=7560c03701fe10683347e72e3a84b287/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:11,069 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:55:11,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7560c03701fe10683347e72e3a84b287; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72116335, jitterRate=0.07461713254451752}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:55:11,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:11,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., pid=42, masterSystemTime=1682362511057 2023-04-24 18:55:11,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,076 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,076 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362511076"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362511076"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362511076"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362511076"}]},"ts":"1682362511076"} 2023-04-24 18:55:11,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-24 18:55:11,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 in 171 msec 2023-04-24 18:55:11,079 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-24 18:55:11,079 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, REOPEN/MOVE in 486 msec 2023-04-24 18:55:11,080 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-24 18:55:11,080 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-24 18:55:11,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-24 18:55:11,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-24 18:55:11,083 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,088 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,088 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,093 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,093 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,096 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,097 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,100 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,100 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,103 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,104 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,108 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,108 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,112 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,112 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,117 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,117 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,120 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,121 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:11,125 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:11,128 DEBUG [hconnection-0x61baffb4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,130 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,130 DEBUG [hconnection-0x693a793e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,130 DEBUG [hconnection-0x4c3cf06d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,132 DEBUG [hconnection-0x4508c72c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,132 DEBUG [hconnection-0x3a68f301-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,133 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,133 DEBUG [hconnection-0x50fdf6dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,136 DEBUG [hconnection-0x6c6d862e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,137 DEBUG [hconnection-0x4873e94f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,137 DEBUG [hconnection-0x147c5dff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,138 DEBUG [hconnection-0x4c07871f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:11,138 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,139 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,140 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,141 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,141 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,142 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,142 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,142 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:11,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,144 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:55:11,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:11,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:11,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:11,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:11,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:11,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:11,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,160 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd82e81[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,174 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,184 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,184 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682362511165, max timestamp=1682362511183Num uniques 859; 2023-04-24 18:55:11,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-24 18:55:11,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362511173, max timestamp=1682362511183Num uniques 859; 2023-04-24 18:55:11,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:11,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362511173, max timestamp=1682362511183Num uniques 859; 2023-04-24 18:55:11,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:11,186 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,186 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,186 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,187 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,191 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,191 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362571197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362571198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362571198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362571199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362571199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,291 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,296 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,298 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362571300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,304 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,304 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,304 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,304 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,310 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362571306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362571307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362571307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362571307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65152a4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,314 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,314 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,314 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,315 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,318 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,318 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:11,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,325 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,325 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,325 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,325 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,325 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,425 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,429 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,430 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62ae8340[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,450 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,450 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,453 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,457 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,457 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362571502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362571511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362571512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362571512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362571512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,557 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,560 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,563 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,563 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f557ff[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,571 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,573 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,574 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,578 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,580 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,580 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:11,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,580 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,583 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a58ebf68e4bb4a069707680722d25572 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a58ebf68e4bb4a069707680722d25572 2023-04-24 18:55:11,588 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/06944c6ea17848cea575c593e03ec3e7 2023-04-24 18:55:11,681 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,685 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,687 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,689 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,689 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,689 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,689 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,691 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db4dd46[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,697 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,701 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,701 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,703 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:11,707 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,707 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,807 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362571807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,811 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,814 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362571814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,816 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,817 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,817 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362571814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,817 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362571815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:11,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362571816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fc7626[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,827 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,829 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,830 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:11,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,836 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,836 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,936 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:11,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:11,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:11,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:11,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:11,939 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:11,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:11,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:11,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,941 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:11,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:11,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:11,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:11,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:11,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:11,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:11,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:11,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:11,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:11,943 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:11,945 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,946 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,946 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775e4dda[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:11,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:11,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:11,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:11,955 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:11,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:11,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:11,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:11,955 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:11,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:11,959 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:11,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:11,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:11,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:11,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:11,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:11,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:11,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:11,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:11,964 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:11,964 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:11,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:11,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/8cdcae3c5bc742169e740904369f0ae6 2023-04-24 18:55:12,064 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:12,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:12,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:12,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:12,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:12,068 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:12,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:12,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:12,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,070 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:12,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:12,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:12,072 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,072 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:12,072 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:12,072 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:12,073 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:12,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbea661[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:12,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:12,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,078 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:12,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:12,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:12,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:12,082 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:12,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:12,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:12,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:12,083 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:12,084 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,084 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,084 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,086 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:12,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:12,089 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,089 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,089 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:12,089 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,090 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,189 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:12,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:12,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:12,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:12,193 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:12,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:12,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:12,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,194 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:12,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:12,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:12,196 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,197 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:12,197 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:12,197 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:12,200 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ab0e7d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,203 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:12,203 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:12,203 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:12,204 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:12,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:12,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:12,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:12,207 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:12,207 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:12,207 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:12,209 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,209 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,209 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:12,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:12,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:12,218 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:12,218 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:12,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362572312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,318 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:12,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:12,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:12,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362572317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:12,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:12,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362572319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:12,321 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:12,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:12,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:12,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:12,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362572320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362572321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,324 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:12,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:12,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:12,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:12,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:12,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:12,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:12,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:12,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146417c8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:12,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,332 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:12,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:12,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:12,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:12,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:12,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:12,336 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:12,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:12,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:12,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:12,336 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:12,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:12,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,339 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:12,340 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,342 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:12,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:12,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:12,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:12,342 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:12,342 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:12,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b41c3e71d8834881a604104d55a1554b 2023-04-24 18:55:12,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/06944c6ea17848cea575c593e03ec3e7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7 2023-04-24 18:55:12,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7, entries=150, sequenceid=14, filesize=25.1 K 2023-04-24 18:55:12,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/8cdcae3c5bc742169e740904369f0ae6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6 2023-04-24 18:55:12,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6, entries=100, sequenceid=14, filesize=9.3 K 2023-04-24 18:55:12,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b41c3e71d8834881a604104d55a1554b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b 2023-04-24 18:55:12,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b, entries=100, sequenceid=14, filesize=9.3 K 2023-04-24 18:55:12,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for 7560c03701fe10683347e72e3a84b287 in 1270ms, sequenceid=14, compaction requested=false 2023-04-24 18:55:12,424 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-24 18:55:12,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:12,443 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:12,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:12,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:12,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:12,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:12,446 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:12,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:12,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:12,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:12,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:12,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:12,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:12,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:12,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:12,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:12,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:12,451 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=143.13 KB heapSize=400.61 KB 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:12,451 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:12,866 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d66def543d4c46a4897f110f2cad2a4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d66def543d4c46a4897f110f2cad2a4a 2023-04-24 18:55:12,872 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:55:13,273 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/c27d5a85831f467e83b6f1c8e60d6db4 2023-04-24 18:55:13,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:13,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362573329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362573329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362573330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362573332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362573333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362573433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362573435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362573436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362573640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362573641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:13,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362573644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,684 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/2177452c3eaf42b0b81b89b911ea1f27 2023-04-24 18:55:13,694 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/2d2db14769cb46ccb8bd21afc0c340f7 2023-04-24 18:55:13,698 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/c27d5a85831f467e83b6f1c8e60d6db4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4 2023-04-24 18:55:13,701 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4, entries=150, sequenceid=50, filesize=25.1 K 2023-04-24 18:55:13,702 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/2177452c3eaf42b0b81b89b911ea1f27 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27 2023-04-24 18:55:13,706 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27, entries=150, sequenceid=50, filesize=11.6 K 2023-04-24 18:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/2d2db14769cb46ccb8bd21afc0c340f7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7 2023-04-24 18:55:13,725 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7, entries=150, sequenceid=50, filesize=11.6 K 2023-04-24 18:55:13,726 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~400.56 KB/410176, currentSize=40.25 KB/41220 for 7560c03701fe10683347e72e3a84b287 in 1275ms, sequenceid=50, compaction requested=false 2023-04-24 18:55:13,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:13,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:13,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d1b803[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:13,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:13,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,732 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:13,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:13,734 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:13,737 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:13,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:13,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:13,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:13,737 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:13,739 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,739 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,739 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:13,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:13,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:13,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:13,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:13,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:13,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:13,744 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:13,744 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:13,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:13,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,844 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:13,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:13,849 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:13,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:13,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:13,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:13,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:13,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:13,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:13,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:13,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:13,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:13,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:13,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:13,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:13,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:13,853 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:55:13,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:13,853 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:13,854 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:13,854 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:13,854 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:13,854 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:13,889 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d7ab9d72a6304a6fb6f7e5465e8dd338 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7ab9d72a6304a6fb6f7e5465e8dd338 2023-04-24 18:55:13,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:13,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:14,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362514031, max timestamp=1682362514040Num uniques 859; 2023-04-24 18:55:14,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362514031, max timestamp=1682362514040Num uniques 859; 2023-04-24 18:55:14,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:14,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:14,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362514031, max timestamp=1682362514040Num uniques 859; 2023-04-24 18:55:14,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:14,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362574139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362574141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362574142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362574249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362574253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362574254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,302 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=59, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/3ccd6a9ac623496387e5d34dd458d7c7 2023-04-24 18:55:14,382 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/4d6b25d1884e4b368fb5a4839db82e9c 2023-04-24 18:55:14,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362574457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362574462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362574461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362574768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362574770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:14,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362574770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,801 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fc23e9a16c5a4db0ae0640748a686cad 2023-04-24 18:55:14,806 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/3ccd6a9ac623496387e5d34dd458d7c7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7 2023-04-24 18:55:14,810 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7, entries=100, sequenceid=59, filesize=18.4 K 2023-04-24 18:55:14,811 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/4d6b25d1884e4b368fb5a4839db82e9c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c 2023-04-24 18:55:14,814 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c, entries=100, sequenceid=59, filesize=9.3 K 2023-04-24 18:55:14,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fc23e9a16c5a4db0ae0640748a686cad as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad 2023-04-24 18:55:14,818 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad, entries=100, sequenceid=59, filesize=9.3 K 2023-04-24 18:55:14,819 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7560c03701fe10683347e72e3a84b287 in 966ms, sequenceid=59, compaction requested=true 2023-04-24 18:55:14,819 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:14,819 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:14,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:14,821 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397489f6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:14,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:14,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:14,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:14,827 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:14,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:14,830 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:14,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:14,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:14,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:14,830 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:14,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:14,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:14,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:14,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:14,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:14,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:14,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:14,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:14,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:14,840 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:14,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:14,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:14,840 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:14,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:14,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:14,940 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:14,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:14,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:14,944 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:14,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:14,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:14,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:14,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:14,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,945 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:14,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:14,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:14,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:14,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:14,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:14,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:14,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:14,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:14,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:14,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:14,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:14,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:14,947 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:14,947 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:14,947 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:14,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:15,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:15,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362575287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362575288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362575289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362575340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,342 DEBUG [Thread-2314] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362575340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,348 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:15,362 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304244281603f926d4210835c488a2c8a09bc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304244281603f926d4210835c488a2c8a09bc 2023-04-24 18:55:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362575394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362575394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362575395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362575599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362575599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362575599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,768 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/83eb0dbd5b614596a03dcc78b431e88e 2023-04-24 18:55:15,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362575903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362575903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:15,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362575906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,179 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d883252a861d4bbfab8e1b51f9cd2e28 2023-04-24 18:55:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362576407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:16,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362576412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:16,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362576413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,589 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/003cebf6310841beb83f10b5ff0f5a0a 2023-04-24 18:55:16,595 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/83eb0dbd5b614596a03dcc78b431e88e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e 2023-04-24 18:55:16,599 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e, entries=150, sequenceid=95, filesize=25.1 K 2023-04-24 18:55:16,600 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d883252a861d4bbfab8e1b51f9cd2e28 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28 2023-04-24 18:55:16,605 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28, entries=150, sequenceid=95, filesize=11.6 K 2023-04-24 18:55:16,606 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/003cebf6310841beb83f10b5ff0f5a0a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a 2023-04-24 18:55:16,615 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a, entries=150, sequenceid=95, filesize=11.6 K 2023-04-24 18:55:16,616 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 7560c03701fe10683347e72e3a84b287 in 1669ms, sequenceid=95, compaction requested=true 2023-04-24 18:55:16,616 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:16,616 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:16,619 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:16,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7762a043[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:16,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:16,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:16,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:16,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:16,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:16,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:16,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:16,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:16,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:16,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:16,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:16,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:16,631 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:16,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:16,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:16,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:16,631 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:16,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:16,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:16,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:16,637 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,638 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:16,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:16,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:16,640 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:16,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:16,640 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:16,641 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:16,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:16,741 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:16,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:16,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:16,747 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:16,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:16,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:16,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:16,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:16,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:16,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:16,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:16,751 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:16,751 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:16,751 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:16,753 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:16,753 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:16,753 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:55:16,753 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:16,753 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:16,753 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:16,754 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:16,754 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:16,754 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:16,772 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042418d83df8cc554e9cb74a3936a01c8553 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042418d83df8cc554e9cb74a3936a01c8553 2023-04-24 18:55:17,190 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b57d60e87aa5481a80f37828ea09e413 2023-04-24 18:55:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:17,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362517446, max timestamp=1682362517463Num uniques 859; 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362517446, max timestamp=1682362517463Num uniques 859; 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362517446, max timestamp=1682362517463Num uniques 859; 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:17,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:17,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362577501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362577502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362577502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,601 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d141ca35f53c4333a0e3edb860a8e8e3 2023-04-24 18:55:17,611 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/51e9653b398444d59a648093b46cc193 2023-04-24 18:55:17,614 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b57d60e87aa5481a80f37828ea09e413 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413 2023-04-24 18:55:17,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362577610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362577610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362577610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,618 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413, entries=150, sequenceid=105, filesize=25.1 K 2023-04-24 18:55:17,618 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d141ca35f53c4333a0e3edb860a8e8e3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3 2023-04-24 18:55:17,623 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3, entries=150, sequenceid=105, filesize=11.6 K 2023-04-24 18:55:17,623 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/51e9653b398444d59a648093b46cc193 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193 2023-04-24 18:55:17,627 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193, entries=150, sequenceid=105, filesize=11.6 K 2023-04-24 18:55:17,627 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 7560c03701fe10683347e72e3a84b287 in 874ms, sequenceid=105, compaction requested=true 2023-04-24 18:55:17,628 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:17,628 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:17,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:17,629 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,629 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:17,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438c2330[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:17,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:17,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,633 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:17,633 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:17,633 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:17,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,634 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:17,634 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:17,635 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:17,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:17,637 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:17,637 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:17,637 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:17,637 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:17,637 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:17,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,639 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:17,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,639 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:17,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:17,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:17,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:17,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:17,643 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:17,643 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:17,643 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:17,643 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,743 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:17,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:17,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:17,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:17,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:17,746 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:17,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:17,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:17,746 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:17,747 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,747 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:17,749 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:17,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:17,749 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:17,749 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,749 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:17,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:17,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:17,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:17,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:17,750 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:17,750 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:17,751 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:17,751 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:17,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:17,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:17,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362577830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362577831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362577832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362577932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362577940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:17,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:17,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362577940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362578138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362578149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362578149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,166 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304249b5dd76a09af4c34a9af3cbd9bac5c72 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b5dd76a09af4c34a9af3cbd9bac5c72 2023-04-24 18:55:18,173 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/d00a3f55a38946d18662dbe841fda278 2023-04-24 18:55:18,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362578445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362578458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362578458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,587 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/182f8ea21864458cb7ed133676361b9f 2023-04-24 18:55:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362578948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362578969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362578972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:18,998 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fb977f5f97d84b6eab914931ff7a9cf9 2023-04-24 18:55:19,002 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/d00a3f55a38946d18662dbe841fda278 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278 2023-04-24 18:55:19,005 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278, entries=150, sequenceid=142, filesize=25.3 K 2023-04-24 18:55:19,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/182f8ea21864458cb7ed133676361b9f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f 2023-04-24 18:55:19,009 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f, entries=150, sequenceid=142, filesize=11.8 K 2023-04-24 18:55:19,010 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fb977f5f97d84b6eab914931ff7a9cf9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9 2023-04-24 18:55:19,022 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9, entries=150, sequenceid=142, filesize=11.8 K 2023-04-24 18:55:19,023 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 7560c03701fe10683347e72e3a84b287 in 1272ms, sequenceid=142, compaction requested=true 2023-04-24 18:55:19,023 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:19,023 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:19,026 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:19,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:19,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f819ae2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:19,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:19,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,029 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:19,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:19,031 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:19,034 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:19,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:19,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:19,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:19,034 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:19,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:19,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:19,037 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:19,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:19,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:19,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:19,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:19,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:19,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:19,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:19,044 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:19,044 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:19,144 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:19,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:19,152 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:19,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:19,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:19,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:19,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:19,157 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:19,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:19,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:19,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:19,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:19,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:19,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:19,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:19,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:19,161 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:19,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:19,162 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:19,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362519413, max timestamp=1682362519434Num uniques 859; 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362519413, max timestamp=1682362519434Num uniques 859; 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362519413, max timestamp=1682362519434Num uniques 859; 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:19,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362579517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362579523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,578 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424945f9c887de644ac834bf0650956bde5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424945f9c887de644ac834bf0650956bde5 2023-04-24 18:55:19,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362579631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362579640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52196 deadline: 1682362579835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52262 deadline: 1682362579845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362579961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362579977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,983 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/501a6e7d0cc7414d98f6a5d861647e5d 2023-04-24 18:55:19,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362579981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:19,998 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/020a18c3672f4648a5eb89abaecd5ab7 2023-04-24 18:55:20,013 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/718d035cb3794d46b72fae780a215e69 2023-04-24 18:55:20,017 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/501a6e7d0cc7414d98f6a5d861647e5d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d 2023-04-24 18:55:20,020 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d, entries=150, sequenceid=151, filesize=25.3 K 2023-04-24 18:55:20,020 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/020a18c3672f4648a5eb89abaecd5ab7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7 2023-04-24 18:55:20,024 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7, entries=150, sequenceid=151, filesize=11.8 K 2023-04-24 18:55:20,025 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/718d035cb3794d46b72fae780a215e69 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69 2023-04-24 18:55:20,029 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69, entries=150, sequenceid=151, filesize=11.8 K 2023-04-24 18:55:20,029 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7560c03701fe10683347e72e3a84b287 in 868ms, sequenceid=151, compaction requested=true 2023-04-24 18:55:20,029 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:20,029 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,031 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,031 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a793dd[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,035 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,035 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,035 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,040 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,040 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,046 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,046 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,046 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:20,146 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,150 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,152 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362519515, max timestamp=1682362520150Num uniques 1068; 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362519515, max timestamp=1682362520150Num uniques 1068; 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:55:20,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=107.34 KB heapSize=302.36 KB 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362519515, max timestamp=1682362520150Num uniques 1068; 2023-04-24 18:55:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:20,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 18:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:20,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:20,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,157 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,157 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,157 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,157 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,158 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a54372[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,165 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,168 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,172 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,172 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,172 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,172 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,173 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,174 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,179 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,179 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,180 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,180 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,180 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362520234, max timestamp=1682362520277Num uniques 859; 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362520234, max timestamp=1682362520277Num uniques 859; 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362520234, max timestamp=1682362520277Num uniques 859; 2023-04-24 18:55:20,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:20,280 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,283 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,288 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21271f5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,297 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,297 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,298 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,305 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,305 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362520310, max timestamp=1682362520371Num uniques 1068; 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362520310, max timestamp=1682362520371Num uniques 1068; 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362520310, max timestamp=1682362520371Num uniques 1068; 2023-04-24 18:55:20,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:20,405 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,409 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,411 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,413 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,413 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,413 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,413 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,418 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5117addf[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,421 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,422 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,426 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,426 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,433 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,433 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362520443, max timestamp=1682362520480Num uniques 1127; 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362520443, max timestamp=1682362520480Num uniques 1127; 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362520443, max timestamp=1682362520480Num uniques 1127; 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:20,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:20,533 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,538 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,539 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,540 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,541 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,541 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,542 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,543 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,543 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,543 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f50e55b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,558 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,559 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,562 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,562 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,563 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,566 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,566 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,571 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304240ae776a8ccca448abe8f5de7a74e7096 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240ae776a8ccca448abe8f5de7a74e7096 2023-04-24 18:55:20,577 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b01a093a93ac41348771c64433d8cc3c 2023-04-24 18:55:20,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362520540, max timestamp=1682362520578Num uniques 1036; 2023-04-24 18:55:20,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 18:55:20,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362520540, max timestamp=1682362520578Num uniques 1036; 2023-04-24 18:55:20,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 18:55:20,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682362520540, max timestamp=1682362520578Num uniques 1036; 2023-04-24 18:55:20,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 18:55:20,666 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,670 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,673 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,675 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e48d10[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,679 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,683 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,683 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,685 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,689 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,689 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362520663, max timestamp=1682362520695Num uniques 1095; 2023-04-24 18:55:20,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362520663, max timestamp=1682362520695Num uniques 1095; 2023-04-24 18:55:20,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-24 18:55:20,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-24 18:55:20,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682362520663, max timestamp=1682362520695Num uniques 1095; 2023-04-24 18:55:20,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-24 18:55:20,789 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362520733, max timestamp=1682362520794Num uniques 1054; 2023-04-24 18:55:20,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-24 18:55:20,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362520733, max timestamp=1682362520794Num uniques 1054; 2023-04-24 18:55:20,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-24 18:55:20,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682362520733, max timestamp=1682362520794Num uniques 1054; 2023-04-24 18:55:20,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-24 18:55:20,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,805 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,812 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,812 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,812 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,812 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e8cd80[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,819 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,831 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,835 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,835 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362520872, max timestamp=1682362520893Num uniques 1063; 2023-04-24 18:55:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-24 18:55:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362520872, max timestamp=1682362520893Num uniques 1063; 2023-04-24 18:55:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-24 18:55:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682362520872, max timestamp=1682362520893Num uniques 1063; 2023-04-24 18:55:20,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-24 18:55:20,935 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:20,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:20,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:20,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:20,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:20,938 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:20,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:20,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:20,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,940 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:20,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:20,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:20,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:20,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:20,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:20,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:20,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@587df797[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:20,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:20,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:20,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:20,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:20,949 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:20,951 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:20,951 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:20,951 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:20,951 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:20,952 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:20,953 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,953 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:20,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,953 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:20,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,956 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:20,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:20,956 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:20,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:20,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:20,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:20,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:20,957 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:20,957 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:20,957 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:20,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682362520958, max timestamp=1682362520985Num uniques 1022; 2023-04-24 18:55:20,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-24 18:55:20,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682362520958, max timestamp=1682362520985Num uniques 1022; 2023-04-24 18:55:20,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-24 18:55:20,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682362520958, max timestamp=1682362520985Num uniques 1022; 2023-04-24 18:55:20,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-24 18:55:20,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/562597fe28b54239a19c863990e5028a 2023-04-24 18:55:21,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/780ee800d4e84c9c8725641ca4210c09 2023-04-24 18:55:21,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b01a093a93ac41348771c64433d8cc3c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c 2023-04-24 18:55:21,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c, entries=150, sequenceid=189, filesize=25.3 K 2023-04-24 18:55:21,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/562597fe28b54239a19c863990e5028a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a 2023-04-24 18:55:21,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a, entries=150, sequenceid=189, filesize=11.8 K 2023-04-24 18:55:21,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/780ee800d4e84c9c8725641ca4210c09 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09 2023-04-24 18:55:21,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09, entries=150, sequenceid=189, filesize=11.8 K 2023-04-24 18:55:21,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 871ms, sequenceid=189, compaction requested=true 2023-04-24 18:55:21,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:21,026 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:21,026 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-24 18:55:21,026 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-24 18:55:21,026 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:21,026 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:21,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 90850 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-24 18:55:21,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/B is initiating minor compaction (all files) 2023-04-24 18:55:21,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/B in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=88.7 K 2023-04-24 18:55:21,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8cdcae3c5bc742169e740904369f0ae6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682362511151 2023-04-24 18:55:21,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2177452c3eaf42b0b81b89b911ea1f27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682362511185 2023-04-24 18:55:21,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4d6b25d1884e4b368fb5a4839db82e9c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682362513328 2023-04-24 18:55:21,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d883252a861d4bbfab8e1b51f9cd2e28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362514125 2023-04-24 18:55:21,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d141ca35f53c4333a0e3edb860a8e8e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682362515286 2023-04-24 18:55:21,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 182f8ea21864458cb7ed133676361b9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682362517500 2023-04-24 18:55:21,030 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 020a18c3672f4648a5eb89abaecd5ab7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362517819 2023-04-24 18:55:21,031 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 562597fe28b54239a19c863990e5028a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:21,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=67.09 KB heapSize=179.86 KB 2023-04-24 18:55:21,033 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 199466 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-24 18:55:21,033 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/A is initiating minor compaction (all files) 2023-04-24 18:55:21,033 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/A in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:21,033 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=194.8 K 2023-04-24 18:55:21,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:21,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:21,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:21,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:21,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:21,034 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 06944c6ea17848cea575c593e03ec3e7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682362511151 2023-04-24 18:55:21,035 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c27d5a85831f467e83b6f1c8e60d6db4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682362511185 2023-04-24 18:55:21,035 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3ccd6a9ac623496387e5d34dd458d7c7, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682362513328 2023-04-24 18:55:21,036 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 83eb0dbd5b614596a03dcc78b431e88e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362514125 2023-04-24 18:55:21,036 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b57d60e87aa5481a80f37828ea09e413, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682362515286 2023-04-24 18:55:21,037 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d00a3f55a38946d18662dbe841fda278, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682362517500 2023-04-24 18:55:21,037 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 501a6e7d0cc7414d98f6a5d861647e5d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362517819 2023-04-24 18:55:21,037 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b01a093a93ac41348771c64433d8cc3c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:21,053 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#B#compaction#330 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:21,057 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,081 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,082 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,084 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,084 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,090 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#A#compaction#332 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:21,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,093 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/8c1ad79cb71b400ab231d3404ba84df1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8c1ad79cb71b400ab231d3404ba84df1 2023-04-24 18:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,094 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70afbbfe[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,098 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7560c03701fe10683347e72e3a84b287/B of 7560c03701fe10683347e72e3a84b287 into 8c1ad79cb71b400ab231d3404ba84df1(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:21,098 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:21,098 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/B, priority=8, startTime=1682362521026; duration=0sec 2023-04-24 18:55:21,098 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:21,098 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-24 18:55:21,098 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,099 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,100 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,103 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 90850 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-24 18:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,103 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/C is initiating minor compaction (all files) 2023-04-24 18:55:21,103 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/C in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,103 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=88.7 K 2023-04-24 18:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,104 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b41c3e71d8834881a604104d55a1554b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682362511151 2023-04-24 18:55:21,104 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2d2db14769cb46ccb8bd21afc0c340f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682362511185 2023-04-24 18:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,105 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,105 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fc23e9a16c5a4db0ae0640748a686cad, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682362513328 2023-04-24 18:55:21,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 003cebf6310841beb83f10b5ff0f5a0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682362514125 2023-04-24 18:55:21,106 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 51e9653b398444d59a648093b46cc193, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682362515286 2023-04-24 18:55:21,107 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting fb977f5f97d84b6eab914931ff7a9cf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682362517500 2023-04-24 18:55:21,107 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e6200d6d1b5f4a4bb6cc95a0d4fbb616 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e6200d6d1b5f4a4bb6cc95a0d4fbb616 2023-04-24 18:55:21,107 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,107 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 718d035cb3794d46b72fae780a215e69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682362517819 2023-04-24 18:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,109 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 780ee800d4e84c9c8725641ca4210c09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:21,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,116 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,116 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,124 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#C#compaction#333 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:21,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362581117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362581134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,216 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,219 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,221 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,221 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,225 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712224f6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,229 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,233 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,233 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,237 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,242 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,242 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362581235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362581241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,342 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,346 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,347 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,347 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,348 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,348 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,349 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,350 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,350 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,351 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354739f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,355 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,356 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,358 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,358 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,358 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,358 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,358 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,360 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,360 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,360 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,361 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:21,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,365 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,365 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,365 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,365 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,365 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,365 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362581444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362581445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,465 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,469 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,471 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,471 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,472 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,472 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,474 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,474 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,477 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3422fd2d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,479 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,481 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,484 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,484 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,491 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,491 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,491 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,491 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,491 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,491 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,522 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b9a9647ff1d64439a6e986b68cda00e9 2023-04-24 18:55:21,538 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/c57a646323f14525ae00484d5b3c3d8a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c57a646323f14525ae00484d5b3c3d8a 2023-04-24 18:55:21,542 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/5406a717825b44639f47c41311d9cb79 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5406a717825b44639f47c41311d9cb79 2023-04-24 18:55:21,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7560c03701fe10683347e72e3a84b287/A of 7560c03701fe10683347e72e3a84b287 into c57a646323f14525ae00484d5b3c3d8a(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:21,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:21,544 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/A, priority=8, startTime=1682362521026; duration=0sec 2023-04-24 18:55:21,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:21,547 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 7560c03701fe10683347e72e3a84b287/C of 7560c03701fe10683347e72e3a84b287 into 5406a717825b44639f47c41311d9cb79(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:21,547 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:21,547 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/C, priority=8, startTime=1682362521026; duration=0sec 2023-04-24 18:55:21,547 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:21,591 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,594 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,594 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,596 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,596 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,597 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,598 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,598 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,598 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,599 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39dfd01c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,603 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,603 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,603 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,607 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,607 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,613 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,618 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,618 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,719 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,722 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,723 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,724 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,724 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,726 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,727 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593a4645[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,729 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,729 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,731 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,733 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,735 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,735 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,738 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,741 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,742 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,742 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362581756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362581757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,842 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,845 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,846 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,848 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,850 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,852 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:21,854 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79521e76[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:21,854 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:21,855 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:21,855 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:21,855 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:21,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,857 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:21,859 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:21,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:21,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:21,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:21,859 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:21,862 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,862 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,862 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:21,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,866 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,866 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:21,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d5a055f7734741679be8f68e28ddf802 2023-04-24 18:55:21,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/f99d46d9566e469888c1f957f4a1ab8a 2023-04-24 18:55:21,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/b9a9647ff1d64439a6e986b68cda00e9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9 2023-04-24 18:55:21,966 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:21,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:21,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9, entries=250, sequenceid=336, filesize=39.2 K 2023-04-24 18:55:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:21,980 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:21,980 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,980 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:21,980 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:21,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d5a055f7734741679be8f68e28ddf802 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802 2023-04-24 18:55:21,982 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:21,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362581980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:21,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:21,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:21,985 DEBUG [Thread-2312] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:21,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:21,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:21,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:21,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:21,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:21,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802, entries=150, sequenceid=336, filesize=11.9 K 2023-04-24 18:55:21,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,989 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:21,989 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:21,990 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:21,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/f99d46d9566e469888c1f957f4a1ab8a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a 2023-04-24 18:55:21,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:21,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362581992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:21,998 DEBUG [Thread-2318] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3a4fe4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a, entries=150, sequenceid=336, filesize=11.9 K 2023-04-24 18:55:22,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~199.73 KB/204528, currentSize=107.34 KB/109920 for 7560c03701fe10683347e72e3a84b287 in 969ms, sequenceid=336, compaction requested=false 2023-04-24 18:55:22,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:22,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,011 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,013 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,013 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:22,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:22,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:22,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:22,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:22,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:22,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,017 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,017 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,019 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,019 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,019 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,020 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,024 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,024 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,025 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,025 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,025 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,025 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:22,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362582070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,126 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,129 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,129 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,130 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,135 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,136 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,136 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,137 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,137 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,137 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,137 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,140 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da23b8f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,143 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,143 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,144 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,147 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,147 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,149 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,149 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,151 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,155 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,155 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,155 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,155 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362582175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,256 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362582264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362582265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,267 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,267 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,267 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,267 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,268 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,268 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,269 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,269 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,271 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,272 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a15e10[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,280 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,280 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,282 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,283 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,284 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,286 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,286 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,288 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,288 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,288 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,290 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,293 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,294 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:22,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362582384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,394 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,397 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,399 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,401 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,401 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,401 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,401 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,402 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe9531c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,405 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,405 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,412 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,412 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,415 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,417 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,417 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,417 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,418 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,418 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:22,467 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248a017e881690452dbf1779509f6e4181 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248a017e881690452dbf1779509f6e4181 2023-04-24 18:55:22,517 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,523 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,523 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,524 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,529 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,536 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,536 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,537 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a4666a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,539 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,539 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,540 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,540 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,540 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,544 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,544 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,546 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,546 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,546 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:22,551 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,551 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,651 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,657 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,657 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,658 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,659 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,665 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,665 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd64552[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,669 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,675 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,675 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,677 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,678 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,681 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,681 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362582689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,781 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,787 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,788 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,791 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,791 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,791 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,791 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,794 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,795 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@679bd5e6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,806 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,816 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,816 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,882 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/cc9578a98a3142578b4ecea2f37de215 2023-04-24 18:55:22,916 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:22,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:22,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:22,928 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,929 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:22,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:22,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:22,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:22,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:22,938 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525f0bba[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:22,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:22,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,941 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:22,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:22,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,949 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:22,952 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:22,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:22,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:22,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:22,952 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:22,954 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,954 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:22,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,954 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:22,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:22,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:22,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:22,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:22,958 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:22,958 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:22,958 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:22,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:22,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:22,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:22,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,058 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,064 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,071 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ca0ad[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,075 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,078 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,079 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,081 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,083 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,086 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,087 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,087 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,187 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,191 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,193 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:23,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362583197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,200 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa999e9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,203 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,212 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,212 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,212 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,212 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,212 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,217 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,217 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,218 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,222 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,222 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362583272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362583274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/c041364597ce4956b724c796ad85a0d2 2023-04-24 18:55:23,322 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,326 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,332 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,332 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,332 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,332 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,335 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a4511f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,339 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,339 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,339 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,343 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,343 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,343 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,343 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,343 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,350 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,350 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,350 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,359 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,359 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,359 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,459 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,463 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,465 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,465 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,465 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,465 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,467 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,467 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,467 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,467 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,467 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,469 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0c96aa[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,471 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,471 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,472 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,472 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,472 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,474 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,476 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,476 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,478 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,483 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,484 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,484 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,484 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,484 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,584 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,587 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,587 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,588 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,588 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,589 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,589 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,593 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,593 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,593 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,593 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,595 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f764c4e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,597 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,598 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,601 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,601 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,604 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,604 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,606 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,607 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,611 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,611 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,712 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/36bd78a6053b4bdfa1a12271174fd1b3 2023-04-24 18:55:23,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/cc9578a98a3142578b4ecea2f37de215 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215 2023-04-24 18:55:23,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215, entries=200, sequenceid=359, filesize=32.3 K 2023-04-24 18:55:23,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,724 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/c041364597ce4956b724c796ad85a0d2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2 2023-04-24 18:55:23,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,727 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2, entries=150, sequenceid=359, filesize=11.9 K 2023-04-24 18:55:23,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,729 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/36bd78a6053b4bdfa1a12271174fd1b3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,730 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,730 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,730 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:23,730 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:23,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,733 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,733 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3, entries=150, sequenceid=359, filesize=11.9 K 2023-04-24 18:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 1721ms, sequenceid=359, compaction requested=true 2023-04-24 18:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:23,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:23,735 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:23,735 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:23,735 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:23,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c93fc98[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:23,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,735 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:23,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:23,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:23,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:23,737 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,738 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,740 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99386 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:23,740 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:23,740 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/A is initiating minor compaction (all files) 2023-04-24 18:55:23,740 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/B is initiating minor compaction (all files) 2023-04-24 18:55:23,740 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/A in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:23,740 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/B in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:23,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:23,741 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8c1ad79cb71b400ab231d3404ba84df1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=35.8 K 2023-04-24 18:55:23,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:23,741 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c57a646323f14525ae00484d5b3c3d8a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=97.1 K 2023-04-24 18:55:23,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:23,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:23,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:23,741 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8c1ad79cb71b400ab231d3404ba84df1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:23,741 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c57a646323f14525ae00484d5b3c3d8a, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:23,741 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d5a055f7734741679be8f68e28ddf802, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682362520999 2023-04-24 18:55:23,742 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b9a9647ff1d64439a6e986b68cda00e9, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682362520999 2023-04-24 18:55:23,742 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c041364597ce4956b724c796ad85a0d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:23,742 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting cc9578a98a3142578b4ecea2f37de215, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:23,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:23,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,747 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:23,748 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:23,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:23,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:23,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:23,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:23,753 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:23,753 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:23,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:23,771 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#A#compaction#340 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:23,777 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/5329c5d1584c444dafd3a9d0185bef98 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/5329c5d1584c444dafd3a9d0185bef98 2023-04-24 18:55:23,781 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7560c03701fe10683347e72e3a84b287/B of 7560c03701fe10683347e72e3a84b287 into 5329c5d1584c444dafd3a9d0185bef98(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:23,781 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:23,781 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/B, priority=13, startTime=1682362523735; duration=0sec 2023-04-24 18:55:23,781 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:23,781 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:23,782 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:23,782 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/C is initiating minor compaction (all files) 2023-04-24 18:55:23,782 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/C in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,782 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5406a717825b44639f47c41311d9cb79, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=35.8 K 2023-04-24 18:55:23,783 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5406a717825b44639f47c41311d9cb79, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682362519516 2023-04-24 18:55:23,783 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f99d46d9566e469888c1f957f4a1ab8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682362520999 2023-04-24 18:55:23,783 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 36bd78a6053b4bdfa1a12271174fd1b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:23,801 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:23,837 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/20c660600c7b429c854664f55d7be5ca as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/20c660600c7b429c854664f55d7be5ca 2023-04-24 18:55:23,843 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7560c03701fe10683347e72e3a84b287/C of 7560c03701fe10683347e72e3a84b287 into 20c660600c7b429c854664f55d7be5ca(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:23,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:23,844 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/C, priority=13, startTime=1682362523736; duration=0sec 2023-04-24 18:55:23,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:23,853 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:23,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:23,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:23,857 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:23,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:23,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:23,858 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,858 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,859 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:23,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:23,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:23,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:23,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:23,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:23,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:23,878 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:23,878 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:23,878 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:23,878 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:23,878 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:23,879 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:23,905 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248b938987731c4656a5f3e111d5851e7e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b938987731c4656a5f3e111d5851e7e 2023-04-24 18:55:24,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:24,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:24,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362584258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,315 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/fe84e885772144148a73e6aa190bb655 2023-04-24 18:55:24,336 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/515b2762fff74e26b523a1e6ea1e5be5 2023-04-24 18:55:24,354 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/f7eb45ce65af44458092a96abcb6b1af 2023-04-24 18:55:24,358 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/fe84e885772144148a73e6aa190bb655 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655 2023-04-24 18:55:24,362 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655, entries=150, sequenceid=373, filesize=25.4 K 2023-04-24 18:55:24,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:24,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52286 deadline: 1682362584362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,370 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/515b2762fff74e26b523a1e6ea1e5be5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5 2023-04-24 18:55:24,374 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5, entries=150, sequenceid=373, filesize=11.9 K 2023-04-24 18:55:24,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/f7eb45ce65af44458092a96abcb6b1af as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af 2023-04-24 18:55:24,379 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af, entries=150, sequenceid=373, filesize=11.9 K 2023-04-24 18:55:24,380 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7560c03701fe10683347e72e3a84b287 in 502ms, sequenceid=373, compaction requested=false 2023-04-24 18:55:24,380 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:24,380 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:24,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:24,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:24,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:24,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:24,382 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,382 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:24,382 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,382 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:24,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:24,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686b6ebc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:24,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:24,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,386 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:24,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:24,386 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:24,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:24,389 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:24,389 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:24,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:24,392 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:24,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:24,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:24,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:24,392 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:24,394 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,394 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:24,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:24,394 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:24,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:24,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:24,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:24,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:24,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:24,399 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:24,399 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:24,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:24,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:24,399 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:24,499 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:24,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:24,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:24,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:24,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:24,502 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:24,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:24,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:24,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:24,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,503 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:24,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:24,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:24,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:24,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:24,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:24,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:24,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:24,505 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:24,506 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:24,506 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:24,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:24,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:24,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:24,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362584613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,643 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/e018f747409d41529e6273be8e150d16 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/e018f747409d41529e6273be8e150d16 2023-04-24 18:55:24,647 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7560c03701fe10683347e72e3a84b287/A of 7560c03701fe10683347e72e3a84b287 into e018f747409d41529e6273be8e150d16(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:24,647 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:24,647 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/A, priority=13, startTime=1682362523735; duration=0sec 2023-04-24 18:55:24,648 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:24,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362584721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:24,925 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304249ca7d33ef2254d8a9f7dc4dd84e538db to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249ca7d33ef2254d8a9f7dc4dd84e538db 2023-04-24 18:55:24,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362584928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:25,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:25,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362585234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:25,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:25,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362585280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:25,287 DEBUG [Thread-2314] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:25,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362585298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:25,301 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:25,330 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=393, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/431e695008cf4cc79a15dbfe0cccb7e2 2023-04-24 18:55:25,740 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/170b2b36c466492391419bb227aa3d4f 2023-04-24 18:55:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:25,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362585745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52238 deadline: 1682362586006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,014 DEBUG [Thread-2312] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:26,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:26,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52246 deadline: 1682362586020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,022 DEBUG [Thread-2318] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:26,179 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/0208c262813446bdafe6a039a47fc846 2023-04-24 18:55:26,182 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/431e695008cf4cc79a15dbfe0cccb7e2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2 2023-04-24 18:55:26,189 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2, entries=150, sequenceid=393, filesize=25.4 K 2023-04-24 18:55:26,190 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/170b2b36c466492391419bb227aa3d4f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f 2023-04-24 18:55:26,194 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f, entries=150, sequenceid=393, filesize=11.9 K 2023-04-24 18:55:26,195 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/0208c262813446bdafe6a039a47fc846 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846 2023-04-24 18:55:26,199 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846, entries=150, sequenceid=393, filesize=11.9 K 2023-04-24 18:55:26,204 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 1698ms, sequenceid=393, compaction requested=true 2023-04-24 18:55:26,204 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:26,204 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:26,205 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:26,206 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,206 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ea1cdd[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:26,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:26,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,213 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,213 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:26,213 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:26,213 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:26,213 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,213 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:26,214 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:26,214 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,214 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:26,215 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:26,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:26,220 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:26,220 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:26,220 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:26,220 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:26,220 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:26,223 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,223 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,223 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:26,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:26,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:26,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,231 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:26,231 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:26,231 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:26,331 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:26,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:26,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:26,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:26,335 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:26,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:26,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:26,340 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:26,341 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,341 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:26,341 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,374 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304240253de38ce424d41be93236623bc13b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240253de38ce424d41be93236623bc13b3 2023-04-24 18:55:26,391 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=406, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/c0ab1df598c7425fb84843dcbb3ab041 2023-04-24 18:55:26,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:26,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:26,812 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d891b8a833b34591a05e731a1cb09023 2023-04-24 18:55:26,823 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/56e7a3bf1c8f45988c34b56aa9e33ae9 2023-04-24 18:55:26,826 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/c0ab1df598c7425fb84843dcbb3ab041 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041 2023-04-24 18:55:26,830 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041, entries=150, sequenceid=406, filesize=25.4 K 2023-04-24 18:55:26,830 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/d891b8a833b34591a05e731a1cb09023 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023 2023-04-24 18:55:26,834 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023, entries=150, sequenceid=406, filesize=11.9 K 2023-04-24 18:55:26,834 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/56e7a3bf1c8f45988c34b56aa9e33ae9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9 2023-04-24 18:55:26,837 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9, entries=150, sequenceid=406, filesize=11.9 K 2023-04-24 18:55:26,838 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 498ms, sequenceid=406, compaction requested=true 2023-04-24 18:55:26,838 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:26,838 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:26,840 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:26,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:26,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:26,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:26,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d733eb2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:26,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:26,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,846 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:26,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:26,847 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:26,849 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:26,849 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,849 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:26,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:26,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:26,851 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:26,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:26,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:26,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:26,851 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:26,853 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,853 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,853 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:26,854 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:26,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:26,858 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:26,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:26,858 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,858 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:26,858 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,858 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,958 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:26,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:26,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:26,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:26,964 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:26,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:26,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:26,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:26,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,969 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:26,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:26,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:26,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:26,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:26,971 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:26,971 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:26,971 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:26,971 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:26,974 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,974 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:26,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362586969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:26,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e276eb8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:26,977 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:26,977 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:26,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:26,980 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:26,983 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:26,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:26,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:26,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:26,984 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:26,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,987 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:26,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:26,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:26,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:26,992 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:26,992 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:26,992 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:26,992 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:26,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:26,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:26,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:27,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362587078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,093 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,096 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,096 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,097 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,097 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,098 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,098 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,099 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,099 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,099 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,099 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,101 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0ae14e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,102 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,102 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,103 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,105 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,108 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,108 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,108 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,108 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,108 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,110 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,110 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,110 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,120 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,120 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,220 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,224 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,224 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,227 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,230 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0600c6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,233 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,237 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,247 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,247 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,250 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,253 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424fea5605e51734f24809e6ad76fb9a3e7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fea5605e51734f24809e6ad76fb9a3e7 2023-04-24 18:55:27,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,258 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,258 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,258 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:27,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362587284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,358 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,361 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,361 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,362 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,363 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,365 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,365 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,365 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,365 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,369 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,369 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c4317a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,375 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,375 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,375 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,376 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,379 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,379 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,379 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,382 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,384 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,384 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,384 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,385 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,485 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,488 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,491 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,492 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,492 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,492 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c53dc54[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,496 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,502 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,502 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,503 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,503 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,503 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,505 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,510 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,510 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362587591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,610 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,614 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,619 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,622 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,624 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56c94401[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,624 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,625 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,630 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,630 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,632 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,634 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,638 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,638 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,638 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,638 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,659 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/aa09907a814d4d6ba151b9b0e8b2f3b1 2023-04-24 18:55:27,738 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,744 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,744 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,746 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,746 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,749 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,749 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,749 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,749 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b346e2e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,754 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,754 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,754 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,760 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,760 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,764 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,764 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,764 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,772 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,772 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,772 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,773 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,776 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,776 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,876 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:27,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:27,879 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:27,879 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,879 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:27,879 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:27,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,882 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:27,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:27,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:27,883 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:27,884 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,884 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:27,884 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:27,884 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:27,885 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,886 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:27,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3c5d3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:27,887 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,889 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:27,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,889 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:27,890 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:27,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:27,893 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:27,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:27,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:27,893 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:27,893 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:27,894 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,894 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:27,894 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:27,895 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:27,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,898 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:27,899 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:27,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:27,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:27,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:27,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:27,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,899 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:27,899 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:27,899 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:27,999 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,002 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,003 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,006 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,006 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,007 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,007 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,007 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,007 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:28,007 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:28,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:28,009 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,009 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,009 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,009 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:28,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b01881[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:28,011 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:28,011 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:28,013 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,013 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:28,013 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:28,013 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:28,014 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:28,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,014 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:28,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:28,016 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:28,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:28,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:28,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:28,017 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:28,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:28,036 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,037 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:28,037 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/e3f5b33a32274ecfb0b8d894787e7e2d 2023-04-24 18:55:28,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b11e68321b6e456fa10d4951f452961b 2023-04-24 18:55:28,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/aa09907a814d4d6ba151b9b0e8b2f3b1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1 2023-04-24 18:55:28,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1, entries=200, sequenceid=419, filesize=32.3 K 2023-04-24 18:55:28,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/e3f5b33a32274ecfb0b8d894787e7e2d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d 2023-04-24 18:55:28,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d, entries=150, sequenceid=419, filesize=11.9 K 2023-04-24 18:55:28,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b11e68321b6e456fa10d4951f452961b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b 2023-04-24 18:55:28,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b, entries=150, sequenceid=419, filesize=11.9 K 2023-04-24 18:55:28,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7560c03701fe10683347e72e3a84b287 in 1262ms, sequenceid=419, compaction requested=true 2023-04-24 18:55:28,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:28,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:28,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:55:28,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-24 18:55:28,104 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:28,104 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:28,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:28,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/A is initiating minor compaction (all files) 2023-04-24 18:55:28,106 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/A in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,106 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/e018f747409d41529e6273be8e150d16, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=134.4 K 2023-04-24 18:55:28,106 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61306 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:28,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e018f747409d41529e6273be8e150d16, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:28,106 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/C is initiating minor compaction (all files) 2023-04-24 18:55:28,107 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/C in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,107 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/20c660600c7b429c854664f55d7be5ca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=59.9 K 2023-04-24 18:55:28,107 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fe84e885772144148a73e6aa190bb655, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682362522057 2023-04-24 18:55:28,107 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 20c660600c7b429c854664f55d7be5ca, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:28,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f7eb45ce65af44458092a96abcb6b1af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682362522057 2023-04-24 18:55:28,108 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0208c262813446bdafe6a039a47fc846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682362524235 2023-04-24 18:55:28,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:28,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:28,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,109 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 56e7a3bf1c8f45988c34b56aa9e33ae9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682362524573 2023-04-24 18:55:28,109 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 431e695008cf4cc79a15dbfe0cccb7e2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682362524235 2023-04-24 18:55:28,109 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b11e68321b6e456fa10d4951f452961b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:28,110 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c0ab1df598c7425fb84843dcbb3ab041, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682362524573 2023-04-24 18:55:28,110 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting aa09907a814d4d6ba151b9b0e8b2f3b1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:28,133 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#C#compaction#356 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:28,137 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,152 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,152 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,153 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,155 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#A#compaction#355 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,167 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,167 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,167 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,167 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:28,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:28,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc5b7fa[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:28,173 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:28,173 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,175 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:28,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,175 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:28,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:28,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:28,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:28,179 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:28,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:28,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:28,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:28,179 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:28,183 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,183 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,184 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,185 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:28,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:28,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:28,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362588190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,195 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:28,195 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:28,202 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/7262939d841d447da842a287901c626a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/7262939d841d447da842a287901c626a 2023-04-24 18:55:28,206 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/A of 7560c03701fe10683347e72e3a84b287 into 7262939d841d447da842a287901c626a(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:28,206 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:28,206 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/A, priority=11, startTime=1682362528103; duration=0sec 2023-04-24 18:55:28,206 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:28,206 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:28,207 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61306 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:28,207 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/B is initiating minor compaction (all files) 2023-04-24 18:55:28,207 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/B in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,207 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/5329c5d1584c444dafd3a9d0185bef98, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=59.9 K 2023-04-24 18:55:28,208 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5329c5d1584c444dafd3a9d0185bef98, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682362521114 2023-04-24 18:55:28,208 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 515b2762fff74e26b523a1e6ea1e5be5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682362522057 2023-04-24 18:55:28,208 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 170b2b36c466492391419bb227aa3d4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682362524235 2023-04-24 18:55:28,209 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d891b8a833b34591a05e731a1cb09023, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682362524573 2023-04-24 18:55:28,209 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e3f5b33a32274ecfb0b8d894787e7e2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:28,216 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#B#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:28,295 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,298 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:28,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362588297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,303 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,303 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,303 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,304 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:28,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4508f572[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:28,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:28,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:28,308 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,308 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:28,310 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,310 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:28,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:28,313 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:28,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:28,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:28,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:28,313 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:28,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,316 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,316 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,316 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,319 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:28,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:28,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,320 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:28,320 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:28,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,320 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,320 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,320 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,320 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,420 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,423 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,424 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:28,435 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:28,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e85d5d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:28,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:28,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:28,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:28,440 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:28,444 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:28,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:28,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:28,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:28,445 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:28,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,447 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,447 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,452 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:28,452 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:28,452 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:28,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362588503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,546 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424df43f9314f05412ba136a1c5a83547c7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424df43f9314f05412ba136a1c5a83547c7 2023-04-24 18:55:28,551 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/4eefd29207114dfa9783f09d9130ab19 2023-04-24 18:55:28,552 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,556 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,557 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,559 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,559 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,559 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,559 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/296a28a0409b4528a32aefbfb88c28af 2023-04-24 18:55:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:28,563 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94d32d1[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:28,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:28,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:28,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:28,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:28,569 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/c1b87bdcf2e74233936e9b380c59602f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/c1b87bdcf2e74233936e9b380c59602f 2023-04-24 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:28,572 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:28,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:28,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:28,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:28,572 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:28,574 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/C of 7560c03701fe10683347e72e3a84b287 into c1b87bdcf2e74233936e9b380c59602f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:28,574 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:28,574 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/C, priority=11, startTime=1682362528103; duration=0sec 2023-04-24 18:55:28,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,574 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:28,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:28,575 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:28,575 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/a2280c854653432e86cde2d27761b1cd 2023-04-24 18:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:28,576 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/4eefd29207114dfa9783f09d9130ab19 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19 2023-04-24 18:55:28,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:28,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:28,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:28,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:28,582 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:28,583 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:28,583 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:28,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19, entries=200, sequenceid=439, filesize=32.3 K 2023-04-24 18:55:28,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/296a28a0409b4528a32aefbfb88c28af as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af 2023-04-24 18:55:28,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af, entries=150, sequenceid=439, filesize=11.9 K 2023-04-24 18:55:28,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/a2280c854653432e86cde2d27761b1cd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd 2023-04-24 18:55:28,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd, entries=150, sequenceid=439, filesize=11.9 K 2023-04-24 18:55:28,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 485ms, sequenceid=439, compaction requested=false 2023-04-24 18:55:28,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:28,623 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/4e66243359534d979a76475db2e24dad as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4e66243359534d979a76475db2e24dad 2023-04-24 18:55:28,627 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/B of 7560c03701fe10683347e72e3a84b287 into 4e66243359534d979a76475db2e24dad(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:28,627 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:28,627 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/B, priority=11, startTime=1682362528103; duration=0sec 2023-04-24 18:55:28,627 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:28,683 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:28,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:28,687 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:28,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:28,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:28,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:28,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:28,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:28,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:28,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:28,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:28,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:28,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:28,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:28,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:28,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:28,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:28,694 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:28,694 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:28,694 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:28,695 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:28,710 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304249eb8b3fd406e46f48da66206d6fdead6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249eb8b3fd406e46f48da66206d6fdead6 2023-04-24 18:55:28,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:28,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:28,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362588911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362589013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,115 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/be69be39cbd849dea3e0a5d49728ec68 2023-04-24 18:55:29,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362589219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:29,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52262 deadline: 1682362589287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,293 DEBUG [Thread-2314] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:29,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52196 deadline: 1682362589306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,313 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:29,525 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/59a8cc557ef94bbabe2ec9f76e46c62e 2023-04-24 18:55:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362589525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,533 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/8196111cdff0456aa8cbb70ac0fbcc3c 2023-04-24 18:55:29,536 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/be69be39cbd849dea3e0a5d49728ec68 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68 2023-04-24 18:55:29,540 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68, entries=150, sequenceid=454, filesize=25.4 K 2023-04-24 18:55:29,541 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/59a8cc557ef94bbabe2ec9f76e46c62e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e 2023-04-24 18:55:29,547 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e, entries=150, sequenceid=454, filesize=11.9 K 2023-04-24 18:55:29,547 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/8196111cdff0456aa8cbb70ac0fbcc3c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c 2023-04-24 18:55:29,551 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c, entries=150, sequenceid=454, filesize=11.9 K 2023-04-24 18:55:29,552 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7560c03701fe10683347e72e3a84b287 in 858ms, sequenceid=454, compaction requested=true 2023-04-24 18:55:29,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:29,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:29,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:29,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:29,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d73e65[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:29,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:29,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:29,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:29,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:29,562 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:29,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:29,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:29,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:29,562 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:29,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:29,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:29,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:29,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:29,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:29,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:29,571 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:29,571 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:29,671 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:29,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:29,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:29,674 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:29,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:29,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:29,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:29,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:29,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,677 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:29,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:29,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:29,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:29,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:29,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:29,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:29,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:29,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:29,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:29,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:29,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:29,679 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:29,679 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:29,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:29,691 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d80911f316db4f00856a4e175bd9bb3f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d80911f316db4f00856a4e175bd9bb3f 2023-04-24 18:55:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:30,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362590080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,098 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/d47123be8ec64199b87a639dcb8f476a 2023-04-24 18:55:30,108 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/6cfa09582eff4e269d7cd95b96ea565c 2023-04-24 18:55:30,125 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b420264d160f45ed8892a49527876f1d 2023-04-24 18:55:30,129 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/d47123be8ec64199b87a639dcb8f476a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a 2023-04-24 18:55:30,135 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a, entries=150, sequenceid=474, filesize=25.4 K 2023-04-24 18:55:30,136 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/6cfa09582eff4e269d7cd95b96ea565c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c 2023-04-24 18:55:30,139 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c, entries=150, sequenceid=474, filesize=11.9 K 2023-04-24 18:55:30,140 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/b420264d160f45ed8892a49527876f1d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d 2023-04-24 18:55:30,143 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d, entries=150, sequenceid=474, filesize=11.9 K 2023-04-24 18:55:30,144 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7560c03701fe10683347e72e3a84b287 in 464ms, sequenceid=474, compaction requested=true 2023-04-24 18:55:30,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:30,144 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b76f548[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,154 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,156 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,156 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,160 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,166 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,166 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:30,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:30,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:30,266 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,284 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,286 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:30,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,293 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,293 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,295 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716e4fce[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362590291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,304 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,305 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,308 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,315 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,315 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:30,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362590401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,416 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,419 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,421 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,424 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,424 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,424 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,424 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11552088[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,443 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,443 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,447 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,447 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,447 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,453 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,457 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,457 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,457 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,457 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:30,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,557 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,561 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:30,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,572 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,572 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,572 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,572 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41a100d1[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,575 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,576 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,578 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,581 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,581 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,587 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,587 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,587 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,588 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,589 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,591 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,592 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,592 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,592 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,592 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,593 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,593 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,602 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304245e3c50b0dc184d27a042fb273078619b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e3c50b0dc184d27a042fb273078619b 2023-04-24 18:55:30,607 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=487, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/31306e679dd3404db4dc7565d52ebab1 2023-04-24 18:55:30,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362590610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/6c073e89a07741d5baf6910dac310995 2023-04-24 18:55:30,693 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,696 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,701 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,701 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,701 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,701 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da16118[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,709 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,710 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,713 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,713 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,716 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,719 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,718 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,719 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:30,719 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,719 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,719 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,719 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,819 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,840 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,840 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,840 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,840 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,842 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,842 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,849 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,850 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,860 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,860 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,860 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,860 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,865 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb25d5d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,870 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:30,873 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:30,877 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:30,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:30,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:30,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:30,878 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:30,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,880 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,882 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,885 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:30,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:30,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,885 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,885 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:30,886 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:30,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:30,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:30,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52286 deadline: 1682362590918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,986 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:30,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:30,989 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:30,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:30,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:30,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,989 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,991 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:30,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:30,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,991 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:30,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:30,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:30,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:30,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:30,993 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:30,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26db7f63[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:30,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:30,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,997 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:31,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:31,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:31,002 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:31,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:31,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:31,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:31,002 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:31,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:31,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:31,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,007 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:31,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:31,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:31,007 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:31,007 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:31,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:31,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,008 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:31,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/a159ea9ef61d48a7bc2c4439d25bcf09 2023-04-24 18:55:31,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/31306e679dd3404db4dc7565d52ebab1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1 2023-04-24 18:55:31,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1, entries=200, sequenceid=487, filesize=32.3 K 2023-04-24 18:55:31,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/6c073e89a07741d5baf6910dac310995 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995 2023-04-24 18:55:31,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995, entries=150, sequenceid=487, filesize=11.9 K 2023-04-24 18:55:31,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/a159ea9ef61d48a7bc2c4439d25bcf09 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09 2023-04-24 18:55:31,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09, entries=150, sequenceid=487, filesize=11.9 K 2023-04-24 18:55:31,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7560c03701fe10683347e72e3a84b287 in 869ms, sequenceid=487, compaction requested=true 2023-04-24 18:55:31,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:31,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:31,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:31,059 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:31,059 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:31,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:31,060 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144856 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:31,060 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/A is initiating minor compaction (all files) 2023-04-24 18:55:31,060 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:31,060 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/A in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:31,060 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/B is initiating minor compaction (all files) 2023-04-24 18:55:31,060 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/7262939d841d447da842a287901c626a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=141.5 K 2023-04-24 18:55:31,060 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/B in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:31,061 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4e66243359534d979a76475db2e24dad, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=60.0 K 2023-04-24 18:55:31,061 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7262939d841d447da842a287901c626a, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:31,061 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4e66243359534d979a76475db2e24dad, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:31,061 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4eefd29207114dfa9783f09d9130ab19, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682362526930 2023-04-24 18:55:31,061 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 296a28a0409b4528a32aefbfb88c28af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682362526930 2023-04-24 18:55:31,061 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting be69be39cbd849dea3e0a5d49728ec68, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682362528154 2023-04-24 18:55:31,062 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 59a8cc557ef94bbabe2ec9f76e46c62e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682362528154 2023-04-24 18:55:31,062 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d47123be8ec64199b87a639dcb8f476a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682362528868 2023-04-24 18:55:31,062 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6cfa09582eff4e269d7cd95b96ea565c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682362528868 2023-04-24 18:55:31,062 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 31306e679dd3404db4dc7565d52ebab1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682362530039 2023-04-24 18:55:31,062 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 6c073e89a07741d5baf6910dac310995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682362530039 2023-04-24 18:55:31,073 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:31,077 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#A#compaction#369 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:31,108 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:31,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:31,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:31,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:31,111 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:31,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:31,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:31,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:31,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:31,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:31,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:31,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:31,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:31,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:31,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:31,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:31,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:31,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:31,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:31,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:31,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. started... 2023-04-24 18:55:31,116 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:31,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:31,130 DEBUG [Thread-2329] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:59583 2023-04-24 18:55:31,130 DEBUG [Thread-2329] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,133 DEBUG [Thread-2323] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:59583 2023-04-24 18:55:31,133 DEBUG [Thread-2323] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,134 DEBUG [Thread-2325] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:59583 2023-04-24 18:55:31,134 DEBUG [Thread-2325] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,139 DEBUG [Thread-2321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:59583 2023-04-24 18:55:31,140 DEBUG [Thread-2321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,141 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424bbf63c1c5178423ea26c7cdb857f194f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bbf63c1c5178423ea26c7cdb857f194f 2023-04-24 18:55:31,141 DEBUG [Thread-2327] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:59583 2023-04-24 18:55:31,141 DEBUG [Thread-2327] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:31,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. as already flushing 2023-04-24 18:55:31,424 DEBUG [Thread-2316] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:59583 2023-04-24 18:55:31,424 DEBUG [Thread-2316] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:31,489 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/aa76067cd7664f5c963feeb19da31583 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/aa76067cd7664f5c963feeb19da31583 2023-04-24 18:55:31,493 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/B of 7560c03701fe10683347e72e3a84b287 into aa76067cd7664f5c963feeb19da31583(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:31,493 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:31,493 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/B, priority=11, startTime=1682362531059; duration=0sec 2023-04-24 18:55:31,493 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:31,493 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 18:55:31,494 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 18:55:31,495 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 7560c03701fe10683347e72e3a84b287/C is initiating minor compaction (all files) 2023-04-24 18:55:31,495 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7560c03701fe10683347e72e3a84b287/C in TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:31,495 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/c1b87bdcf2e74233936e9b380c59602f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp, totalSize=60.0 K 2023-04-24 18:55:31,495 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c1b87bdcf2e74233936e9b380c59602f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682362526813 2023-04-24 18:55:31,496 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a2280c854653432e86cde2d27761b1cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1682362526930 2023-04-24 18:55:31,496 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8196111cdff0456aa8cbb70ac0fbcc3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682362528154 2023-04-24 18:55:31,496 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b420264d160f45ed8892a49527876f1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682362528868 2023-04-24 18:55:31,496 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a159ea9ef61d48a7bc2c4439d25bcf09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682362530039 2023-04-24 18:55:31,504 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 7560c03701fe10683347e72e3a84b287#C#compaction#372 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:31,546 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=506, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/8d0e3b67bc334e14ab3b946fe9d94045 2023-04-24 18:55:31,911 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/083c672d57764203aff179eff9df53b2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/083c672d57764203aff179eff9df53b2 2023-04-24 18:55:31,915 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/C of 7560c03701fe10683347e72e3a84b287 into 083c672d57764203aff179eff9df53b2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:31,915 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:31,915 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/C, priority=11, startTime=1682362531059; duration=0sec 2023-04-24 18:55:31,915 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:31,955 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/0508ff0b6d3c416dbf35dbd8c389c60a 2023-04-24 18:55:32,298 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/ff317339f02442a2a7f5903faae88b54 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/ff317339f02442a2a7f5903faae88b54 2023-04-24 18:55:32,301 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 7560c03701fe10683347e72e3a84b287/A of 7560c03701fe10683347e72e3a84b287 into ff317339f02442a2a7f5903faae88b54(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 18:55:32,301 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:32,301 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287., storeName=7560c03701fe10683347e72e3a84b287/A, priority=11, startTime=1682362531059; duration=1sec 2023-04-24 18:55:32,301 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:32,364 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fa9ca18ef9664d209aa5156bba56731e 2023-04-24 18:55:32,368 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/8d0e3b67bc334e14ab3b946fe9d94045 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/8d0e3b67bc334e14ab3b946fe9d94045 2023-04-24 18:55:32,370 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/8d0e3b67bc334e14ab3b946fe9d94045, entries=150, sequenceid=506, filesize=25.4 K 2023-04-24 18:55:32,371 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/0508ff0b6d3c416dbf35dbd8c389c60a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/0508ff0b6d3c416dbf35dbd8c389c60a 2023-04-24 18:55:32,374 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/0508ff0b6d3c416dbf35dbd8c389c60a, entries=150, sequenceid=506, filesize=11.9 K 2023-04-24 18:55:32,375 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/fa9ca18ef9664d209aa5156bba56731e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fa9ca18ef9664d209aa5156bba56731e 2023-04-24 18:55:32,378 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fa9ca18ef9664d209aa5156bba56731e, entries=150, sequenceid=506, filesize=11.9 K 2023-04-24 18:55:32,378 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=6.71 KB/6870 for 7560c03701fe10683347e72e3a84b287 in 1262ms, sequenceid=506, compaction requested=false 2023-04-24 18:55:32,378 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:32,378 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f3544e1[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:32,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:32,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:32,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:32,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:32,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:32,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,385 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:32,385 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:32,385 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:32,387 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:32,387 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:32,387 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:32,387 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:32,387 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:32,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,389 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:32,389 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:32,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:32,389 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:32,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:32,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:32,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:32,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:32,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:32,392 DEBUG [Thread-2320] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:32,392 DEBUG [Thread-2320] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:32,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:32,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:32,493 DEBUG [Thread-2320] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:32,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:36,042 DEBUG [Thread-2312] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:59583 2023-04-24 18:55:36,042 DEBUG [Thread-2312] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:36,044 DEBUG [Thread-2318] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:59583 2023-04-24 18:55:36,044 DEBUG [Thread-2318] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:37,049 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c57a646323f14525ae00484d5b3c3d8a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/e018f747409d41529e6273be8e150d16, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/7262939d841d447da842a287901c626a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1] to archive 2023-04-24 18:55:37,050 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:37,051 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/06944c6ea17848cea575c593e03ec3e7 2023-04-24 18:55:37,052 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c27d5a85831f467e83b6f1c8e60d6db4 2023-04-24 18:55:37,053 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/3ccd6a9ac623496387e5d34dd458d7c7 2023-04-24 18:55:37,054 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/83eb0dbd5b614596a03dcc78b431e88e 2023-04-24 18:55:37,055 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b57d60e87aa5481a80f37828ea09e413 2023-04-24 18:55:37,056 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d00a3f55a38946d18662dbe841fda278 2023-04-24 18:55:37,057 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/501a6e7d0cc7414d98f6a5d861647e5d 2023-04-24 18:55:37,058 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c57a646323f14525ae00484d5b3c3d8a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c57a646323f14525ae00484d5b3c3d8a 2023-04-24 18:55:37,059 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b01a093a93ac41348771c64433d8cc3c 2023-04-24 18:55:37,060 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/b9a9647ff1d64439a6e986b68cda00e9 2023-04-24 18:55:37,061 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/cc9578a98a3142578b4ecea2f37de215 2023-04-24 18:55:37,061 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/e018f747409d41529e6273be8e150d16 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/e018f747409d41529e6273be8e150d16 2023-04-24 18:55:37,062 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/fe84e885772144148a73e6aa190bb655 2023-04-24 18:55:37,063 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/431e695008cf4cc79a15dbfe0cccb7e2 2023-04-24 18:55:37,064 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/c0ab1df598c7425fb84843dcbb3ab041 2023-04-24 18:55:37,065 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/aa09907a814d4d6ba151b9b0e8b2f3b1 2023-04-24 18:55:37,066 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/7262939d841d447da842a287901c626a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/7262939d841d447da842a287901c626a 2023-04-24 18:55:37,067 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/4eefd29207114dfa9783f09d9130ab19 2023-04-24 18:55:37,068 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/be69be39cbd849dea3e0a5d49728ec68 2023-04-24 18:55:37,069 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/d47123be8ec64199b87a639dcb8f476a 2023-04-24 18:55:37,070 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/31306e679dd3404db4dc7565d52ebab1 2023-04-24 18:55:37,073 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8c1ad79cb71b400ab231d3404ba84df1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/5329c5d1584c444dafd3a9d0185bef98, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4e66243359534d979a76475db2e24dad, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995] to archive 2023-04-24 18:55:37,074 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:37,075 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8cdcae3c5bc742169e740904369f0ae6 2023-04-24 18:55:37,076 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/2177452c3eaf42b0b81b89b911ea1f27 2023-04-24 18:55:37,077 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4d6b25d1884e4b368fb5a4839db82e9c 2023-04-24 18:55:37,078 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d883252a861d4bbfab8e1b51f9cd2e28 2023-04-24 18:55:37,078 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d141ca35f53c4333a0e3edb860a8e8e3 2023-04-24 18:55:37,079 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/182f8ea21864458cb7ed133676361b9f 2023-04-24 18:55:37,080 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/020a18c3672f4648a5eb89abaecd5ab7 2023-04-24 18:55:37,081 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8c1ad79cb71b400ab231d3404ba84df1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/8c1ad79cb71b400ab231d3404ba84df1 2023-04-24 18:55:37,081 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/562597fe28b54239a19c863990e5028a 2023-04-24 18:55:37,082 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d5a055f7734741679be8f68e28ddf802 2023-04-24 18:55:37,083 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/5329c5d1584c444dafd3a9d0185bef98 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/5329c5d1584c444dafd3a9d0185bef98 2023-04-24 18:55:37,083 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/c041364597ce4956b724c796ad85a0d2 2023-04-24 18:55:37,084 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/515b2762fff74e26b523a1e6ea1e5be5 2023-04-24 18:55:37,085 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/170b2b36c466492391419bb227aa3d4f 2023-04-24 18:55:37,086 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/d891b8a833b34591a05e731a1cb09023 2023-04-24 18:55:37,086 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4e66243359534d979a76475db2e24dad to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/4e66243359534d979a76475db2e24dad 2023-04-24 18:55:37,087 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/e3f5b33a32274ecfb0b8d894787e7e2d 2023-04-24 18:55:37,088 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/296a28a0409b4528a32aefbfb88c28af 2023-04-24 18:55:37,088 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/59a8cc557ef94bbabe2ec9f76e46c62e 2023-04-24 18:55:37,089 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6cfa09582eff4e269d7cd95b96ea565c 2023-04-24 18:55:37,090 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/6c073e89a07741d5baf6910dac310995 2023-04-24 18:55:37,093 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5406a717825b44639f47c41311d9cb79, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/20c660600c7b429c854664f55d7be5ca, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/c1b87bdcf2e74233936e9b380c59602f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09] to archive 2023-04-24 18:55:37,094 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:55:37,095 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b41c3e71d8834881a604104d55a1554b 2023-04-24 18:55:37,095 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/2d2db14769cb46ccb8bd21afc0c340f7 2023-04-24 18:55:37,096 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fc23e9a16c5a4db0ae0640748a686cad 2023-04-24 18:55:37,097 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/003cebf6310841beb83f10b5ff0f5a0a 2023-04-24 18:55:37,097 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/51e9653b398444d59a648093b46cc193 2023-04-24 18:55:37,098 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fb977f5f97d84b6eab914931ff7a9cf9 2023-04-24 18:55:37,099 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/718d035cb3794d46b72fae780a215e69 2023-04-24 18:55:37,100 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5406a717825b44639f47c41311d9cb79 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5406a717825b44639f47c41311d9cb79 2023-04-24 18:55:37,100 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/780ee800d4e84c9c8725641ca4210c09 2023-04-24 18:55:37,101 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f99d46d9566e469888c1f957f4a1ab8a 2023-04-24 18:55:37,102 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/20c660600c7b429c854664f55d7be5ca to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/20c660600c7b429c854664f55d7be5ca 2023-04-24 18:55:37,103 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/36bd78a6053b4bdfa1a12271174fd1b3 2023-04-24 18:55:37,104 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/f7eb45ce65af44458092a96abcb6b1af 2023-04-24 18:55:37,104 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/0208c262813446bdafe6a039a47fc846 2023-04-24 18:55:37,105 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/56e7a3bf1c8f45988c34b56aa9e33ae9 2023-04-24 18:55:37,106 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/c1b87bdcf2e74233936e9b380c59602f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/c1b87bdcf2e74233936e9b380c59602f 2023-04-24 18:55:37,107 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b11e68321b6e456fa10d4951f452961b 2023-04-24 18:55:37,108 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a2280c854653432e86cde2d27761b1cd 2023-04-24 18:55:37,109 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/8196111cdff0456aa8cbb70ac0fbcc3c 2023-04-24 18:55:37,109 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/b420264d160f45ed8892a49527876f1d 2023-04-24 18:55:37,110 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/a159ea9ef61d48a7bc2c4439d25bcf09 2023-04-24 18:55:39,351 DEBUG [Thread-2310] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:59583 2023-04-24 18:55:39,351 DEBUG [Thread-2310] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:39,374 DEBUG [Thread-2314] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:59583 2023-04-24 18:55:39,374 DEBUG [Thread-2314] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:39,374 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 150 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1885 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5650 rows 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1886 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5657 rows 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1889 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5666 rows 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1885 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5654 rows 2023-04-24 18:55:39,375 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1881 2023-04-24 18:55:39,376 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5642 rows 2023-04-24 18:55:39,376 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:55:39,376 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:59583 2023-04-24 18:55:39,376 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:55:39,380 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:55:39,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:55:39,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:39,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 18:55:39,385 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362539384"}]},"ts":"1682362539384"} 2023-04-24 18:55:39,385 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:55:39,387 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:55:39,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, UNASSIGN}] 2023-04-24 18:55:39,388 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, UNASSIGN 2023-04-24 18:55:39,389 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:39,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362539389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362539389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362539389"}]},"ts":"1682362539389"} 2023-04-24 18:55:39,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 18:55:39,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:39,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7560c03701fe10683347e72e3a84b287, disabling compactions & flushes 2023-04-24 18:55:39,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:39,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:39,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. after waiting 0 ms 2023-04-24 18:55:39,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:39,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7560c03701fe10683347e72e3a84b287 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=A 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=B 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7560c03701fe10683347e72e3a84b287, store=C 2023-04-24 18:55:39,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:39,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304242c1522b3c3d44b2b826ea3f192840587 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242c1522b3c3d44b2b826ea3f192840587 2023-04-24 18:55:39,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/dc53a603ddc043eba92b158b7a9ad340 2023-04-24 18:55:39,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 18:55:39,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/79f55484cb0541d8a78f3b98046a9ad8 2023-04-24 18:55:39,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 18:55:39,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/5c149b07fe184f5eadfcc001593665b3 2023-04-24 18:55:40,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/A/dc53a603ddc043eba92b158b7a9ad340 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/dc53a603ddc043eba92b158b7a9ad340 2023-04-24 18:55:40,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/dc53a603ddc043eba92b158b7a9ad340, entries=100, sequenceid=517, filesize=18.6 K 2023-04-24 18:55:40,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/B/79f55484cb0541d8a78f3b98046a9ad8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/79f55484cb0541d8a78f3b98046a9ad8 2023-04-24 18:55:40,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/79f55484cb0541d8a78f3b98046a9ad8, entries=100, sequenceid=517, filesize=9.5 K 2023-04-24 18:55:40,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/.tmp/C/5c149b07fe184f5eadfcc001593665b3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5c149b07fe184f5eadfcc001593665b3 2023-04-24 18:55:40,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5c149b07fe184f5eadfcc001593665b3, entries=100, sequenceid=517, filesize=9.5 K 2023-04-24 18:55:40,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 7560c03701fe10683347e72e3a84b287 in 469ms, sequenceid=517, compaction requested=true 2023-04-24 18:55:40,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits/520.seqid, newMaxSeqId=520, maxSeqId=4 2023-04-24 18:55:40,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287. 2023-04-24 18:55:40,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7560c03701fe10683347e72e3a84b287: 2023-04-24 18:55:40,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:40,020 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7560c03701fe10683347e72e3a84b287, regionState=CLOSED 2023-04-24 18:55:40,020 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362540020"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362540020"}]},"ts":"1682362540020"} 2023-04-24 18:55:40,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-24 18:55:40,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 7560c03701fe10683347e72e3a84b287, server=jenkins-hbase4.apache.org,37593,1682362415680 in 631 msec 2023-04-24 18:55:40,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-24 18:55:40,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7560c03701fe10683347e72e3a84b287, UNASSIGN in 635 msec 2023-04-24 18:55:40,024 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362540024"}]},"ts":"1682362540024"} 2023-04-24 18:55:40,024 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:55:40,026 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:55:40,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 646 msec 2023-04-24 18:55:40,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 18:55:40,489 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-24 18:55:40,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:55:40,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,492 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,493 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-24 18:55:40,497 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:40,499 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits] 2023-04-24 18:55:40,502 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/8d0e3b67bc334e14ab3b946fe9d94045 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/8d0e3b67bc334e14ab3b946fe9d94045 2023-04-24 18:55:40,504 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/dc53a603ddc043eba92b158b7a9ad340 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/dc53a603ddc043eba92b158b7a9ad340 2023-04-24 18:55:40,505 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/ff317339f02442a2a7f5903faae88b54 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/A/ff317339f02442a2a7f5903faae88b54 2023-04-24 18:55:40,508 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/0508ff0b6d3c416dbf35dbd8c389c60a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/0508ff0b6d3c416dbf35dbd8c389c60a 2023-04-24 18:55:40,509 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/79f55484cb0541d8a78f3b98046a9ad8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/79f55484cb0541d8a78f3b98046a9ad8 2023-04-24 18:55:40,510 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/aa76067cd7664f5c963feeb19da31583 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/B/aa76067cd7664f5c963feeb19da31583 2023-04-24 18:55:40,513 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/083c672d57764203aff179eff9df53b2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/083c672d57764203aff179eff9df53b2 2023-04-24 18:55:40,514 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5c149b07fe184f5eadfcc001593665b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/5c149b07fe184f5eadfcc001593665b3 2023-04-24 18:55:40,516 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fa9ca18ef9664d209aa5156bba56731e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/C/fa9ca18ef9664d209aa5156bba56731e 2023-04-24 18:55:40,523 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits/520.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287/recovered.edits/520.seqid 2023-04-24 18:55:40,524 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/7560c03701fe10683347e72e3a84b287 2023-04-24 18:55:40,524 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:55:40,524 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:55:40,525 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 18:55:40,527 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240253de38ce424d41be93236623bc13b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240253de38ce424d41be93236623bc13b3 2023-04-24 18:55:40,529 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240ae776a8ccca448abe8f5de7a74e7096 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240ae776a8ccca448abe8f5de7a74e7096 2023-04-24 18:55:40,530 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042418d83df8cc554e9cb74a3936a01c8553 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042418d83df8cc554e9cb74a3936a01c8553 2023-04-24 18:55:40,531 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242c1522b3c3d44b2b826ea3f192840587 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242c1522b3c3d44b2b826ea3f192840587 2023-04-24 18:55:40,532 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304244281603f926d4210835c488a2c8a09bc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304244281603f926d4210835c488a2c8a09bc 2023-04-24 18:55:40,533 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e3c50b0dc184d27a042fb273078619b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e3c50b0dc184d27a042fb273078619b 2023-04-24 18:55:40,534 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248a017e881690452dbf1779509f6e4181 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248a017e881690452dbf1779509f6e4181 2023-04-24 18:55:40,536 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b938987731c4656a5f3e111d5851e7e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248b938987731c4656a5f3e111d5851e7e 2023-04-24 18:55:40,537 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424945f9c887de644ac834bf0650956bde5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424945f9c887de644ac834bf0650956bde5 2023-04-24 18:55:40,538 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b5dd76a09af4c34a9af3cbd9bac5c72 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b5dd76a09af4c34a9af3cbd9bac5c72 2023-04-24 18:55:40,539 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249ca7d33ef2254d8a9f7dc4dd84e538db to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249ca7d33ef2254d8a9f7dc4dd84e538db 2023-04-24 18:55:40,540 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249eb8b3fd406e46f48da66206d6fdead6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249eb8b3fd406e46f48da66206d6fdead6 2023-04-24 18:55:40,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a58ebf68e4bb4a069707680722d25572 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a58ebf68e4bb4a069707680722d25572 2023-04-24 18:55:40,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bbf63c1c5178423ea26c7cdb857f194f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bbf63c1c5178423ea26c7cdb857f194f 2023-04-24 18:55:40,544 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d66def543d4c46a4897f110f2cad2a4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d66def543d4c46a4897f110f2cad2a4a 2023-04-24 18:55:40,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7ab9d72a6304a6fb6f7e5465e8dd338 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7ab9d72a6304a6fb6f7e5465e8dd338 2023-04-24 18:55:40,546 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d80911f316db4f00856a4e175bd9bb3f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d80911f316db4f00856a4e175bd9bb3f 2023-04-24 18:55:40,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424df43f9314f05412ba136a1c5a83547c7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424df43f9314f05412ba136a1c5a83547c7 2023-04-24 18:55:40,549 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e6200d6d1b5f4a4bb6cc95a0d4fbb616 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e6200d6d1b5f4a4bb6cc95a0d4fbb616 2023-04-24 18:55:40,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fea5605e51734f24809e6ad76fb9a3e7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fea5605e51734f24809e6ad76fb9a3e7 2023-04-24 18:55:40,550 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:55:40,552 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,553 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:55:40,555 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:55:40,556 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,556 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:55:40,556 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362540556"}]},"ts":"9223372036854775807"} 2023-04-24 18:55:40,558 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:55:40,558 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7560c03701fe10683347e72e3a84b287, NAME => 'TestAcidGuarantees,,1682362508058.7560c03701fe10683347e72e3a84b287.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:55:40,558 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:55:40,558 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362540558"}]},"ts":"9223372036854775807"} 2023-04-24 18:55:40,559 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:55:40,561 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 72 msec 2023-04-24 18:55:40,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-24 18:55:40,595 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-24 18:55:40,608 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 226) - Thread LEAK? -, OpenFileDescriptor=535 (was 558), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=388 (was 371) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4571 (was 4220) - AvailableMemoryMB LEAK? - 2023-04-24 18:55:40,620 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=535, MaxFileDescriptor=60000, SystemLoadAverage=388, ProcessCount=170, AvailableMemoryMB=4575 2023-04-24 18:55:40,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:55:40,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:55:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:55:40,625 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:55:40,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-24 18:55:40,626 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:55:40,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 18:55:40,627 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:40,628 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 empty. 2023-04-24 18:55:40,628 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:40,628 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:55:40,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 18:55:40,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 18:55:41,036 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:55:41,037 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 55e964c86d584c4edcab7a055c74c603, NAME => 'TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:55:41,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:55:41,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 55e964c86d584c4edcab7a055c74c603, disabling compactions & flushes 2023-04-24 18:55:41,049 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,049 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,049 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. after waiting 0 ms 2023-04-24 18:55:41,049 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,049 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,049 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:41,051 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:55:41,051 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362541051"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362541051"}]},"ts":"1682362541051"} 2023-04-24 18:55:41,052 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:55:41,053 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:55:41,053 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362541053"}]},"ts":"1682362541053"} 2023-04-24 18:55:41,054 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:55:41,057 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, ASSIGN}] 2023-04-24 18:55:41,059 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, ASSIGN 2023-04-24 18:55:41,059 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:55:41,210 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=55e964c86d584c4edcab7a055c74c603, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,211 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362541210"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362541210"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362541210"}]},"ts":"1682362541210"} 2023-04-24 18:55:41,212 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:55:41,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 18:55:41,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 55e964c86d584c4edcab7a055c74c603, NAME => 'TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:55:41,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:55:41,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,367 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,369 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A 2023-04-24 18:55:41,369 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A 2023-04-24 18:55:41,369 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:41,369 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 55e964c86d584c4edcab7a055c74c603 columnFamilyName A 2023-04-24 18:55:41,369 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(310): Store=55e964c86d584c4edcab7a055c74c603/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:41,370 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,371 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B 2023-04-24 18:55:41,371 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B 2023-04-24 18:55:41,371 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:41,371 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 55e964c86d584c4edcab7a055c74c603 columnFamilyName B 2023-04-24 18:55:41,371 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(310): Store=55e964c86d584c4edcab7a055c74c603/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:41,371 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,372 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C 2023-04-24 18:55:41,373 DEBUG [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C 2023-04-24 18:55:41,373 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:55:41,373 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 55e964c86d584c4edcab7a055c74c603 columnFamilyName C 2023-04-24 18:55:41,373 INFO [StoreOpener-55e964c86d584c4edcab7a055c74c603-1] regionserver.HStore(310): Store=55e964c86d584c4edcab7a055c74c603/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:55:41,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:55:41,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:55:41,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 55e964c86d584c4edcab7a055c74c603; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70477685, jitterRate=0.050199344754219055}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:55:41,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:41,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., pid=49, masterSystemTime=1682362541363 2023-04-24 18:55:41,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,381 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=55e964c86d584c4edcab7a055c74c603, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,381 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362541381"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362541381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362541381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362541381"}]},"ts":"1682362541381"} 2023-04-24 18:55:41,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-24 18:55:41,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 in 170 msec 2023-04-24 18:55:41,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-24 18:55:41,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, ASSIGN in 326 msec 2023-04-24 18:55:41,385 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:55:41,385 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362541385"}]},"ts":"1682362541385"} 2023-04-24 18:55:41,386 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:55:41,388 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:55:41,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 765 msec 2023-04-24 18:55:41,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 18:55:41,730 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-24 18:55:41,731 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,735 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,737 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,738 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,739 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:55:41,741 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:55:41,742 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,746 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,747 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,750 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,751 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,756 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,756 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,762 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,763 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,767 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,768 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,775 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,775 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,784 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,784 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,788 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,789 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,799 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,799 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:55:41,807 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:55:41,812 DEBUG [hconnection-0xe8f310d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,825 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,825 DEBUG [hconnection-0x36b53294-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,825 DEBUG [hconnection-0x40c18922-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,825 DEBUG [hconnection-0x75d8e8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,825 DEBUG [hconnection-0x1112132a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,827 DEBUG [hconnection-0x63f3d22e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,827 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,827 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,827 DEBUG [hconnection-0x70c43878-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,828 DEBUG [hconnection-0x2cdd4b0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,828 DEBUG [hconnection-0x78e58414-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,831 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,833 DEBUG [hconnection-0x2ac1f732-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:55:41,835 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,835 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,835 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,837 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,837 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,837 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:55:41,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:41,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:41,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:41,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:41,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362541861, max timestamp=1682362541865Num uniques 859; 2023-04-24 18:55:41,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362541861, max timestamp=1682362541865Num uniques 859; 2023-04-24 18:55:41,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:41,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:41,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362541861, max timestamp=1682362541865Num uniques 859; 2023-04-24 18:55:41,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362601873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362601872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362601874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362601874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362601874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:41,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:41,880 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:41,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:41,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:41,880 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:41,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:41,883 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:41,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:41,883 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:41,883 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,883 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:41,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:41,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:41,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:41,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:41,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:41,892 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:41,894 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:41,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:41,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:41,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790a51be[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:41,896 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,897 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:41,898 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,898 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:41,900 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:41,900 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:41,902 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:41,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:41,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:41,902 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:41,902 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:41,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:41,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:41,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:41,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:41,906 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:41,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:41,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:41,910 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:41,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:41,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:41,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:41,910 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:41,910 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:41,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:41,910 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:41,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:41,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:41,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,010 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:42,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:42,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:42,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:42,022 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:42,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:42,022 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:42,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,023 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,024 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,024 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,024 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:42,027 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:42,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:42,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:42,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:42,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:42,034 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,034 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:42,034 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:42,034 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755eb377[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:42,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:42,038 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:42,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:42,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:42,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:42,045 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:42,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:42,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:42,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:42,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:42,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,049 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,050 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,050 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,052 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:42,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:42,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,053 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:42,053 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:42,153 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:42,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:42,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:42,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:42,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:42,157 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:42,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:42,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:42,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:42,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:42,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:42,172 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,172 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:42,172 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:42,172 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:42,176 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,177 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:42,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@803fad8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:42,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:42,179 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:42,181 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:42,181 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:42,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:42,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362602178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362602178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362602179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,182 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:42,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:42,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362602180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,185 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:42,186 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:42,186 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:42,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362602181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:42,186 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:42,186 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:42,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,189 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:42,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:42,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,196 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:42,196 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:42,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/2a2c42f266414fe3b0ea1fd669c6539c 2023-04-24 18:55:42,296 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:42,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:42,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:42,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:42,301 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:42,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:42,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:42,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,304 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:42,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:42,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:42,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:42,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:42,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:42,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:42,312 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:42,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:42,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:42,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80 2023-04-24 18:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,319 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,319 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d54a575[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:42,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:42,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,326 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:42,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:42,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,328 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:42,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:42,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:42,331 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:42,331 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:42,331 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:42,331 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:42,331 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:42,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,334 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:42,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:42,337 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,337 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,338 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:42,338 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/d65103e22b0d4dbf884736942d71ffb3 2023-04-24 18:55:42,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:42,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/2a2c42f266414fe3b0ea1fd669c6539c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c 2023-04-24 18:55:42,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:55:42,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80 2023-04-24 18:55:42,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:55:42,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/d65103e22b0d4dbf884736942d71ffb3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3 2023-04-24 18:55:42,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 18:55:42,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 55e964c86d584c4edcab7a055c74c603 in 517ms, sequenceid=12, compaction requested=false 2023-04-24 18:55:42,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:42,438 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:42,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:42,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:42,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:42,441 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:42,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:42,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:42,442 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,442 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:42,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:42,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:42,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:42,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:42,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:42,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:42,447 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:42,447 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:42,447 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:42,462 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/092d62c1f96248efade4a9e5a59b3c28 2023-04-24 18:55:42,473 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/efaa7857960a48e09bea672bbff37e4a 2023-04-24 18:55:42,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:42,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:42,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362602495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362602501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362602504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362602505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362602505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362602606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362602606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362602613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362602613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362602616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362602812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362602813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362602819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362602819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362602827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,893 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a9d51e396d0e4ae6bcef57a7848b3b40 2023-04-24 18:55:42,898 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/092d62c1f96248efade4a9e5a59b3c28 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28 2023-04-24 18:55:42,902 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:55:42,903 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/efaa7857960a48e09bea672bbff37e4a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a 2023-04-24 18:55:42,907 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:55:42,908 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a9d51e396d0e4ae6bcef57a7848b3b40 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40 2023-04-24 18:55:42,913 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 18:55:42,914 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 55e964c86d584c4edcab7a055c74c603 in 467ms, sequenceid=48, compaction requested=false 2023-04-24 18:55:42,914 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:42,914 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,917 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:42,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f88ac0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:42,920 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:42,920 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,921 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:42,921 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,921 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:42,922 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:42,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:42,927 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:42,927 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:42,927 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:42,927 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:42,927 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:42,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:42,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:42,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:42,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:42,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:42,936 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:42,936 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:42,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:42,936 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:42,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,036 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:43,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:43,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:43,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:43,043 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:43,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:43,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:43,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,047 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:43,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:43,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:43,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:43,055 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:43,055 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:43,055 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:43,055 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,056 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:43,056 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,070 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/0383718c919140c6b69a13f3fc2caa71 2023-04-24 18:55:43,092 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/f73e2848f088491881c6be3f74786807 2023-04-24 18:55:43,112 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/76f51b0b57e7432280b5dcf3fa3b8539 2023-04-24 18:55:43,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/0383718c919140c6b69a13f3fc2caa71 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71 2023-04-24 18:55:43,121 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71, entries=150, sequenceid=60, filesize=11.6 K 2023-04-24 18:55:43,123 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/f73e2848f088491881c6be3f74786807 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807 2023-04-24 18:55:43,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:43,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:43,129 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807, entries=150, sequenceid=60, filesize=11.6 K 2023-04-24 18:55:43,130 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/76f51b0b57e7432280b5dcf3fa3b8539 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539 2023-04-24 18:55:43,140 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539, entries=150, sequenceid=60, filesize=11.6 K 2023-04-24 18:55:43,142 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=73.80 KB/75570 for 55e964c86d584c4edcab7a055c74c603 in 87ms, sequenceid=60, compaction requested=true 2023-04-24 18:55:43,142 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:43,142 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:43,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:43,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:43,144 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:43,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:43,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bd4863[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:43,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:43,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,149 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:43,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,149 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:43,153 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:43,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,153 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:43,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:43,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:43,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:43,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:43,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:43,158 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:43,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,160 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,161 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:43,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:43,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,167 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:43,167 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:43,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362603167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/210c2868b0604595b3f7975830faa2c4 2023-04-24 18:55:43,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362603169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362603170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362603173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362603174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,267 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:43,270 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:43,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:43,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:43,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,275 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:43,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:43,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:43,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:43,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:43,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:43,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:43,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:43,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:43,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:43,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:43,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,281 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,281 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717fbb56[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:43,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:43,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362603279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362603284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,286 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362603284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:43,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:43,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:43,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:43,288 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:43,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:43,291 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:43,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:43,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:43,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:43,291 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:43,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362603288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362603288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,311 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:43,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:43,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:43,315 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:43,315 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:43,415 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:43,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:43,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:43,420 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:43,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:43,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:43,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,422 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,422 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,422 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,422 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:43,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:43,426 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,426 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:43,426 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:43,426 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:43,427 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e8d701[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:43,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:43,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:43,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:43,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:43,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:43,436 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,437 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:43,443 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:43,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:43,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:43,443 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:43,443 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:43,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,453 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:43,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:43,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:43,453 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:43,453 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:43,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362603487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362603487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362603488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362603499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362603500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,553 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:43,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:43,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:43,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:43,557 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:43,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:43,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:43,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:43,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:43,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:43,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:43,562 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:43,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:43,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:43,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:43,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be4f076[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:43,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:43,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:43,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:43,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:43,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:43,573 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:43,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:43,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:43,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:43,573 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:43,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:43,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,582 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:43,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:43,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:43,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,585 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:43,585 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:43,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:43,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:43,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/9c83c99e3ca9499d8125279c428bdbc4 2023-04-24 18:55:43,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/53b5c6ff78d34ffb9632a1649605c529 2023-04-24 18:55:43,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/210c2868b0604595b3f7975830faa2c4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4 2023-04-24 18:55:43,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4, entries=250, sequenceid=75, filesize=16.2 K 2023-04-24 18:55:43,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/9c83c99e3ca9499d8125279c428bdbc4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4 2023-04-24 18:55:43,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4, entries=150, sequenceid=75, filesize=11.6 K 2023-04-24 18:55:43,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/53b5c6ff78d34ffb9632a1649605c529 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529 2023-04-24 18:55:43,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529, entries=150, sequenceid=75, filesize=11.6 K 2023-04-24 18:55:43,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 55e964c86d584c4edcab7a055c74c603 in 495ms, sequenceid=75, compaction requested=true 2023-04-24 18:55:43,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:43,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:43,639 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:55:43,639 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:55:43,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:43,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:43,640 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:55:43,640 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:55:43,640 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:43,640 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:43,641 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,641 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,641 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=51 K 2023-04-24 18:55:43,641 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=46.4 K 2023-04-24 18:55:43,641 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2a2c42f266414fe3b0ea1fd669c6539c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362541840 2023-04-24 18:55:43,641 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9ecf0f054eae4b2f9fb3eeb94ba6ef80, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362541840 2023-04-24 18:55:43,641 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 092d62c1f96248efade4a9e5a59b3c28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362541871 2023-04-24 18:55:43,641 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting efaa7857960a48e09bea672bbff37e4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362541871 2023-04-24 18:55:43,642 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0383718c919140c6b69a13f3fc2caa71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682362542493 2023-04-24 18:55:43,642 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f73e2848f088491881c6be3f74786807, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682362542493 2023-04-24 18:55:43,642 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 210c2868b0604595b3f7975830faa2c4, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543129 2023-04-24 18:55:43,642 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9c83c99e3ca9499d8125279c428bdbc4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543137 2023-04-24 18:55:43,654 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:43,654 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:43,683 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5345ee885edc4ff58a6812c5caf14f48 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5345ee885edc4ff58a6812c5caf14f48 2023-04-24 18:55:43,685 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:43,688 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 5345ee885edc4ff58a6812c5caf14f48(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:43,688 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:43,688 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=12, startTime=1682362543639; duration=0sec 2023-04-24 18:55:43,688 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:43,688 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:55:43,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:43,692 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:55:43,692 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:43,692 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,692 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=46.4 K 2023-04-24 18:55:43,693 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d65103e22b0d4dbf884736942d71ffb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682362541840 2023-04-24 18:55:43,693 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a9d51e396d0e4ae6bcef57a7848b3b40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682362541871 2023-04-24 18:55:43,694 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 76f51b0b57e7432280b5dcf3fa3b8539, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682362542493 2023-04-24 18:55:43,694 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 53b5c6ff78d34ffb9632a1649605c529, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543137 2023-04-24 18:55:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:43,696 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:43,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:43,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:43,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:43,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:43,699 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:43,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:43,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:43,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:43,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:43,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:43,704 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:43,704 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:43,704 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 18:55:43,704 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:43,704 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,705 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:43,705 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,705 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:43,705 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:43,714 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#392 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:43,734 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/e68bbaabb307411a9714504a0d31cfc2 2023-04-24 18:55:43,741 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/7a73927c56074037948bddd509845b2b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7a73927c56074037948bddd509845b2b 2023-04-24 18:55:43,748 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 7a73927c56074037948bddd509845b2b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:43,748 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:43,748 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=12, startTime=1682362543639; duration=0sec 2023-04-24 18:55:43,748 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:43,760 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/b681adfee4594dfb894ff215e4da4995 2023-04-24 18:55:43,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:43,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:43,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362603818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362603818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362603819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362603821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362603822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362603925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362603925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362603926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362603931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:43,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362603931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,074 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/360a3bd576084c1285a6020d8e75d10d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/360a3bd576084c1285a6020d8e75d10d 2023-04-24 18:55:44,078 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 360a3bd576084c1285a6020d8e75d10d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:44,078 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:44,078 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=12, startTime=1682362543639; duration=0sec 2023-04-24 18:55:44,078 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:44,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362604130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362604131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362604131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362604138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362604139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,175 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a242746ac46a4ba69f7f26ba81ede1c3 2023-04-24 18:55:44,179 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/e68bbaabb307411a9714504a0d31cfc2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2 2023-04-24 18:55:44,182 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2, entries=150, sequenceid=93, filesize=11.6 K 2023-04-24 18:55:44,183 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/b681adfee4594dfb894ff215e4da4995 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995 2023-04-24 18:55:44,189 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995, entries=150, sequenceid=93, filesize=11.6 K 2023-04-24 18:55:44,190 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a242746ac46a4ba69f7f26ba81ede1c3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3 2023-04-24 18:55:44,194 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3, entries=150, sequenceid=93, filesize=11.6 K 2023-04-24 18:55:44,194 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 55e964c86d584c4edcab7a055c74c603 in 490ms, sequenceid=93, compaction requested=false 2023-04-24 18:55:44,194 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:44,194 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:44,194 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:44,197 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37164939[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:44,199 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:44,199 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,200 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:44,201 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,201 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:44,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:44,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:44,205 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:44,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:44,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:44,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:44,205 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:44,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,213 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,213 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,214 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:44,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:44,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:44,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:44,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:44,214 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:44,214 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,314 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:44,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:44,319 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:44,319 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:44,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:44,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:44,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,321 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:44,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:44,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:44,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:44,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:44,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:44,323 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:44,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:44,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:44,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:44,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362604459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362604460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362604462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362604463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362604464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362604566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362604566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362604572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362604573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362604573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,742 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f3ae041b9dfa49c49cdc2af8cf9f7d13 2023-04-24 18:55:44,753 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/843c09208ffd4f7590107be5dc21e05e 2023-04-24 18:55:44,764 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/cc6ef7a27ed84609b16e153fdc31650b 2023-04-24 18:55:44,771 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f3ae041b9dfa49c49cdc2af8cf9f7d13 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13 2023-04-24 18:55:44,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362604772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362604773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,777 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13, entries=150, sequenceid=110, filesize=11.6 K 2023-04-24 18:55:44,778 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/843c09208ffd4f7590107be5dc21e05e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e 2023-04-24 18:55:44,783 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e, entries=150, sequenceid=110, filesize=11.6 K 2023-04-24 18:55:44,784 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/cc6ef7a27ed84609b16e153fdc31650b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b 2023-04-24 18:55:44,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362604777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362604779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362604780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,788 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b, entries=150, sequenceid=110, filesize=11.6 K 2023-04-24 18:55:44,789 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 55e964c86d584c4edcab7a055c74c603 in 466ms, sequenceid=110, compaction requested=true 2023-04-24 18:55:44,789 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:44,789 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:44,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:44,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:44,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:44,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbff186[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:44,793 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:44,793 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:44,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:44,797 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:44,797 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:44,800 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:44,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:44,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:44,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:44,800 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:44,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,802 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:44,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:44,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:44,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:44,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:44,807 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:44,807 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:44,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:44,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:44,907 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:44,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:44,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:44,911 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:44,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:44,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:44,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:44,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,912 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:44,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:44,913 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:44,913 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:44,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:44,914 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:44,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:44,915 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:44,915 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:44,916 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:44,916 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:45,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:45,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362605099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362605100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362605104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362605107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362605107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362605209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362605210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362605210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362605220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362605220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,334 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/c4acd94010894c5fb9209eaf201fb01f 2023-04-24 18:55:45,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362605411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362605415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362605415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362605426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362605426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,550 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:55:45,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362605717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362605719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362605719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362605729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362605731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,745 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/17ce62562d6a49f19736503530080635 2023-04-24 18:55:45,757 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/fa0f9bdc057943469784f5f0c021b7f1 2023-04-24 18:55:45,762 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/c4acd94010894c5fb9209eaf201fb01f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f 2023-04-24 18:55:45,766 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f, entries=150, sequenceid=127, filesize=11.6 K 2023-04-24 18:55:45,767 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/17ce62562d6a49f19736503530080635 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635 2023-04-24 18:55:45,775 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635, entries=150, sequenceid=127, filesize=11.6 K 2023-04-24 18:55:45,776 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/fa0f9bdc057943469784f5f0c021b7f1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1 2023-04-24 18:55:45,780 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1, entries=150, sequenceid=127, filesize=11.6 K 2023-04-24 18:55:45,781 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 55e964c86d584c4edcab7a055c74c603 in 866ms, sequenceid=127, compaction requested=true 2023-04-24 18:55:45,781 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:45,781 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:45,788 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b0692f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:45,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:45,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:45,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:45,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:45,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:45,794 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:45,801 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:45,801 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:45,801 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:45,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:45,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,807 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:45,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:45,810 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:45,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:45,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,817 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:45,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,818 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:45,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:45,817 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:45,818 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:45,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:45,918 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:45,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:45,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:45,922 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:45,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:45,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:45,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:45,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:45,923 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:45,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:45,924 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:45,924 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:45,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:45,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:45,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:45,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:45,927 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:45,927 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:45,927 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:46,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362606246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362606251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362606252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362606258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362606260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,347 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/692a1b5c3c0e4a42b7a8fb995724b19b 2023-04-24 18:55:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362606359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362606359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362606360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362606361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362606368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362606568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362606568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362606568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362606568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362606577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,758 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/f89555fd3af54ca59bcdde10a83eb84c 2023-04-24 18:55:46,772 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ebf593451b3f4fc4928ebfbf3e36d83d 2023-04-24 18:55:46,775 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/692a1b5c3c0e4a42b7a8fb995724b19b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b 2023-04-24 18:55:46,779 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b, entries=150, sequenceid=143, filesize=11.8 K 2023-04-24 18:55:46,779 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/f89555fd3af54ca59bcdde10a83eb84c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c 2023-04-24 18:55:46,782 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c, entries=150, sequenceid=143, filesize=11.8 K 2023-04-24 18:55:46,783 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ebf593451b3f4fc4928ebfbf3e36d83d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d 2023-04-24 18:55:46,786 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d, entries=150, sequenceid=143, filesize=11.8 K 2023-04-24 18:55:46,786 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 55e964c86d584c4edcab7a055c74c603 in 859ms, sequenceid=143, compaction requested=true 2023-04-24 18:55:46,786 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:46,787 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:46,788 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784b528f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:46,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:46,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:46,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:46,793 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:46,795 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:46,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:46,795 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:46,798 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,798 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:46,798 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:46,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,810 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:46,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:46,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:46,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,812 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:46,812 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:46,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:46,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:46,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:46,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362606891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362606894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,912 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362606905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362606905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362606906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:46,916 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:46,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:46,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:46,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:46,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,916 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,919 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:46,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:46,919 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,919 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:46,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:46,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:46,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:46,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:46,920 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,922 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d11833e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:46,925 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:46,925 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:46,926 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,926 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:46,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:46,929 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:46,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:46,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:46,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:46,930 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:46,935 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:46,935 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:46,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,936 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,938 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,938 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:46,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:46,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:46,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:46,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:46,938 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:46,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:46,939 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:46,939 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:46,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362607006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362607008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362607014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362607014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362607014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,039 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,043 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,043 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,044 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,044 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,045 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,045 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,046 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,048 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,049 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39667edb[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,050 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,050 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,053 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,053 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,060 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,063 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,063 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,063 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,063 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,063 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,065 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,067 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,074 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,074 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,174 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,177 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,183 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,183 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,183 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,183 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,185 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef891c9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,187 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,187 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,188 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,190 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,190 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,190 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,194 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,195 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,195 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,196 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,197 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,200 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:47,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,201 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,201 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,201 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,202 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362607213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362607215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362607220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362607220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362607220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/314eb857867546cb9ad6ed73dc3a1471 2023-04-24 18:55:47,301 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,304 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,308 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,308 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,309 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,309 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,311 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c056fb3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,316 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,316 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,316 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,320 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,321 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,325 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,328 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,329 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,329 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,429 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,433 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,435 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,435 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,436 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfc1b7e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,446 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,446 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,452 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,452 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,452 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362607518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362607521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362607524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362607525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362607526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,553 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,556 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7663cbc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,571 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,571 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,571 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,575 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,579 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,579 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,680 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,683 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,685 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8a8340[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/0bf3a82d736c4b57883f6dbda9d1f352 2023-04-24 18:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,698 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,698 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,705 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,706 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,706 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,707 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,707 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:47,807 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,810 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,815 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa50fb5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,822 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,827 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,831 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:47,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,834 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,835 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,835 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,935 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:47,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:47,938 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:47,938 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:47,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:47,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,940 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:47,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:47,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:47,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:47,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:47,942 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:47,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:47,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:47,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27efca74[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:47,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:47,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,948 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:47,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:47,952 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,956 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:47,956 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,956 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:47,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:47,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:47,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:47,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:47,958 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:47,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:47,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:47,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:47,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:47,964 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:47,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:47,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:47,965 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:47,965 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:47,965 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:47,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362608030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362608030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362608030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362608032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362608035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,065 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,070 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,070 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,071 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,073 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,075 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,075 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,075 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:48,075 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,078 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11da91e9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,082 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,082 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,087 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,087 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,089 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,089 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,089 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,090 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,095 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,095 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,095 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/143a91236c3a4b03b1a2f081dcd5901b 2023-04-24 18:55:48,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/314eb857867546cb9ad6ed73dc3a1471 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471 2023-04-24 18:55:48,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471, entries=250, sequenceid=162, filesize=16.4 K 2023-04-24 18:55:48,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/0bf3a82d736c4b57883f6dbda9d1f352 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352 2023-04-24 18:55:48,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352, entries=150, sequenceid=162, filesize=11.8 K 2023-04-24 18:55:48,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/143a91236c3a4b03b1a2f081dcd5901b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b 2023-04-24 18:55:48,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b, entries=150, sequenceid=162, filesize=11.8 K 2023-04-24 18:55:48,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 55e964c86d584c4edcab7a055c74c603 in 1252ms, sequenceid=162, compaction requested=true 2023-04-24 18:55:48,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:48,130 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:55:48,130 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:55:48,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:48,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:48,133 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76534 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:55:48,133 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71754 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:55:48,134 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:48,134 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:48,134 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,134 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,134 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/360a3bd576084c1285a6020d8e75d10d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=74.7 K 2023-04-24 18:55:48,134 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5345ee885edc4ff58a6812c5caf14f48, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=70.1 K 2023-04-24 18:55:48,134 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 360a3bd576084c1285a6020d8e75d10d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543137 2023-04-24 18:55:48,134 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5345ee885edc4ff58a6812c5caf14f48, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543137 2023-04-24 18:55:48,134 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e68bbaabb307411a9714504a0d31cfc2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682362543154 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b681adfee4594dfb894ff215e4da4995, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682362543154 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f3ae041b9dfa49c49cdc2af8cf9f7d13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682362543817 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 843c09208ffd4f7590107be5dc21e05e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682362543817 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c4acd94010894c5fb9209eaf201fb01f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362544453 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 17ce62562d6a49f19736503530080635, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362544453 2023-04-24 18:55:48,135 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 692a1b5c3c0e4a42b7a8fb995724b19b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682362545094 2023-04-24 18:55:48,136 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f89555fd3af54ca59bcdde10a83eb84c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682362545094 2023-04-24 18:55:48,136 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 314eb857867546cb9ad6ed73dc3a1471, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:48,136 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0bf3a82d736c4b57883f6dbda9d1f352, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:48,154 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#408 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:48,160 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#409 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:48,171 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/b37e5bce274248739f8e095a88c9745d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b37e5bce274248739f8e095a88c9745d 2023-04-24 18:55:48,177 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/b2fb23dcce19424b97ca2f144dc16971 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b2fb23dcce19424b97ca2f144dc16971 2023-04-24 18:55:48,177 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into b37e5bce274248739f8e095a88c9745d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:48,177 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,177 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=10, startTime=1682362548130; duration=0sec 2023-04-24 18:55:48,177 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:48,177 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:55:48,180 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71754 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:55:48,180 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:48,180 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,180 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7a73927c56074037948bddd509845b2b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=70.1 K 2023-04-24 18:55:48,181 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7a73927c56074037948bddd509845b2b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682362543137 2023-04-24 18:55:48,181 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a242746ac46a4ba69f7f26ba81ede1c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682362543154 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cc6ef7a27ed84609b16e153fdc31650b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1682362543817 2023-04-24 18:55:48,182 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into b2fb23dcce19424b97ca2f144dc16971(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fa0f9bdc057943469784f5f0c021b7f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682362544453 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,182 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=10, startTime=1682362548130; duration=0sec 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting ebf593451b3f4fc4928ebfbf3e36d83d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682362545094 2023-04-24 18:55:48,182 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 143a91236c3a4b03b1a2f081dcd5901b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:48,190 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:48,195 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,204 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,206 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,206 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,211 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,211 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,211 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,218 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:48,219 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:48,225 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/f3b9e12e75724adca3538e065e9a9a23 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/f3b9e12e75724adca3538e065e9a9a23 2023-04-24 18:55:48,232 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/19c1db999cfe4bdfae1af08846db7ff3 2023-04-24 18:55:48,233 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into f3b9e12e75724adca3538e065e9a9a23(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:48,233 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,233 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=10, startTime=1682362548130; duration=0sec 2023-04-24 18:55:48,233 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:48,258 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5f51c559be964197b1cff3e3025c1ee1 2023-04-24 18:55:48,275 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/1756e5678f1d439b9b1a411042bdb014 2023-04-24 18:55:48,280 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/19c1db999cfe4bdfae1af08846db7ff3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3 2023-04-24 18:55:48,284 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3, entries=150, sequenceid=178, filesize=11.8 K 2023-04-24 18:55:48,285 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5f51c559be964197b1cff3e3025c1ee1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1 2023-04-24 18:55:48,288 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1, entries=150, sequenceid=178, filesize=11.8 K 2023-04-24 18:55:48,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/1756e5678f1d439b9b1a411042bdb014 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014 2023-04-24 18:55:48,293 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014, entries=150, sequenceid=178, filesize=11.8 K 2023-04-24 18:55:48,293 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 55e964c86d584c4edcab7a055c74c603 in 75ms, sequenceid=178, compaction requested=false 2023-04-24 18:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58039011[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,300 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,303 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,305 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,305 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,311 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,315 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,315 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,415 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,419 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,421 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,423 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,423 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,423 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,423 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,425 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab9f403[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,430 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,434 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,434 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,445 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,445 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,545 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,549 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,550 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,553 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,553 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,553 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635c354a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,561 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,566 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,566 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,572 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,572 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,672 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,676 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,676 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,676 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,676 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,678 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,678 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,680 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,682 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,682 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,682 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,682 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,684 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@264ca511[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,686 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,687 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,692 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,692 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,700 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,700 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,800 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,804 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,805 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,806 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,808 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,808 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,808 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,809 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,810 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620b2ad1[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,814 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,816 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,818 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,818 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,820 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,820 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,829 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,829 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,929 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:48,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:48,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:48,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:48,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:48,933 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:48,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:48,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:48,934 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,934 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,935 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:48,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:48,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:48,939 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,939 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:48,939 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:48,939 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,941 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1767d37f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:48,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:48,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,945 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:48,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,945 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:48,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,951 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:48,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:48,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:48,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:48,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:48,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:48,955 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,955 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:48,956 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:48,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:48,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:48,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:48,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,960 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:48,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:48,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:48,960 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:48,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:48,961 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:49,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:55:49,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:49,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:49,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:49,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,061 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,064 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,065 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,065 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,066 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,067 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,068 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,072 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159342bc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362609070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,075 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,077 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,077 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362609070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362609071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362609072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,079 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,079 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362609073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,080 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,080 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,082 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,085 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,086 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:49,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362609175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362609179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362609179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362609180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362609180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,186 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,189 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,189 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,189 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,189 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,189 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,194 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56340932[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,198 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,199 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,201 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,205 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,205 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,207 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,208 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,214 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,214 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,215 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,215 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,315 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,318 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,319 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,319 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,326 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d52ec87[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,330 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,333 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,333 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,335 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,339 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,339 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,340 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,339 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,340 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,340 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,340 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362609381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362609385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362609386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362609386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362609387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,440 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,444 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,448 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44872679[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,453 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,453 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,454 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/d278520ca88f44c792cf99bee93cb0f0 2023-04-24 18:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,461 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,461 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,461 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,461 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,461 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,462 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,463 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,463 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,465 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,469 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,473 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,474 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,474 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,474 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,475 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,475 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,575 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,578 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,578 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,580 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,580 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,582 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,582 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,582 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,582 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8e1af7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,588 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,588 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,589 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,591 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,591 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,591 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,595 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,595 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,595 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,597 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,597 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,597 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,598 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,601 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,604 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,605 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,605 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,605 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362609686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362609691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362609692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362609693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:49,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362609693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,705 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,708 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,708 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,710 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,710 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,710 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,711 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,712 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,712 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,712 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d85f72d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,717 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,717 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,719 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,719 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,719 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,719 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,721 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,723 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,723 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,723 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,724 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,724 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,724 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,726 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,728 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,728 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,728 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,728 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,828 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,832 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,833 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,836 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,836 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,837 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,838 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,838 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:49,838 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:49,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,840 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c912ce[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:49,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:49,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:49,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,845 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:49,848 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:49,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:49,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:49,848 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:49,848 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:49,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:49,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,851 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:49,852 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,853 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:49,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:49,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:49,856 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:49,857 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:49,857 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:49,857 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:49,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/e417bff71e8f4903856d3046d9428e47 2023-04-24 18:55:49,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/178150e6bf854af2b22cd29e56d47267 2023-04-24 18:55:49,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/d278520ca88f44c792cf99bee93cb0f0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0 2023-04-24 18:55:49,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0, entries=300, sequenceid=196, filesize=18.8 K 2023-04-24 18:55:49,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/e417bff71e8f4903856d3046d9428e47 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47 2023-04-24 18:55:49,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47, entries=150, sequenceid=196, filesize=11.8 K 2023-04-24 18:55:49,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/178150e6bf854af2b22cd29e56d47267 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267 2023-04-24 18:55:49,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267, entries=150, sequenceid=196, filesize=11.8 K 2023-04-24 18:55:49,903 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 55e964c86d584c4edcab7a055c74c603 in 858ms, sequenceid=196, compaction requested=true 2023-04-24 18:55:49,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:49,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:49,903 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:49,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:49,903 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:49,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:49,904 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43620 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:49,904 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:49,904 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:49,904 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:49,904 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,904 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,904 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b37e5bce274248739f8e095a88c9745d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=42.6 K 2023-04-24 18:55:49,904 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b2fb23dcce19424b97ca2f144dc16971, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=35.6 K 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b37e5bce274248739f8e095a88c9745d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b2fb23dcce19424b97ca2f144dc16971, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 19c1db999cfe4bdfae1af08846db7ff3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682362546889 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5f51c559be964197b1cff3e3025c1ee1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682362546889 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d278520ca88f44c792cf99bee93cb0f0, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:49,905 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e417bff71e8f4903856d3046d9428e47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:49,912 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#417 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:49,912 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#418 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:49,928 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/9487a087659c4ce3b9dbf4506d60c60c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/9487a087659c4ce3b9dbf4506d60c60c 2023-04-24 18:55:49,932 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 9487a087659c4ce3b9dbf4506d60c60c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:49,933 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:49,933 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362549903; duration=0sec 2023-04-24 18:55:49,933 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:49,933 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:49,933 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:49,934 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:49,934 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,934 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/f3b9e12e75724adca3538e065e9a9a23, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=35.6 K 2023-04-24 18:55:49,934 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f3b9e12e75724adca3538e065e9a9a23, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682362546250 2023-04-24 18:55:49,934 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1756e5678f1d439b9b1a411042bdb014, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682362546889 2023-04-24 18:55:49,935 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 178150e6bf854af2b22cd29e56d47267, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:49,941 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#419 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:49,957 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:49,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:49,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:49,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:49,960 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:49,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:49,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:49,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:49,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:49,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:49,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:49,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:49,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:49,968 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:49,968 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:49,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:49,983 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/77f25a29fad5426eb01181bac56a50c2 2023-04-24 18:55:50,000 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/b827a5293b504b53aa27e407d1223083 2023-04-24 18:55:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:50,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:50,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362610205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362610207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362610208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362610214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362610215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362610316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362610318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362610318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362610321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,326 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/7127a1e5e1644c4bbc8091a398eeb001 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7127a1e5e1644c4bbc8091a398eeb001 2023-04-24 18:55:50,330 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 7127a1e5e1644c4bbc8091a398eeb001(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:50,330 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,330 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362549903; duration=0sec 2023-04-24 18:55:50,330 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:50,353 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e2d475364fc44fa588cdb5dd606b0f60 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e2d475364fc44fa588cdb5dd606b0f60 2023-04-24 18:55:50,359 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into e2d475364fc44fa588cdb5dd606b0f60(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:50,359 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,359 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362549903; duration=0sec 2023-04-24 18:55:50,359 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:50,415 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/be332a0c4ab8469dab94d0b18573aaa0 2023-04-24 18:55:50,419 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/77f25a29fad5426eb01181bac56a50c2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2 2023-04-24 18:55:50,423 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2, entries=150, sequenceid=217, filesize=11.8 K 2023-04-24 18:55:50,423 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/b827a5293b504b53aa27e407d1223083 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083 2023-04-24 18:55:50,427 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083, entries=150, sequenceid=217, filesize=11.8 K 2023-04-24 18:55:50,428 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/be332a0c4ab8469dab94d0b18573aaa0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0 2023-04-24 18:55:50,431 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0, entries=150, sequenceid=217, filesize=11.8 K 2023-04-24 18:55:50,432 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 55e964c86d584c4edcab7a055c74c603 in 464ms, sequenceid=217, compaction requested=false 2023-04-24 18:55:50,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:50,435 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,435 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8040a0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:50,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:50,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,440 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:50,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:50,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:50,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,440 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:50,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:50,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:50,444 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:50,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:50,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:50,444 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:50,444 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:50,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:50,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,451 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,451 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,452 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:50,452 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:50,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:50,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:50,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:55:50,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:50,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:50,530 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,530 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:50,530 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/1ea8879e58b44e5ea55e0057e7690fc5 2023-04-24 18:55:50,552 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:50,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/35c2eddee62d42539e81c7206423eeae 2023-04-24 18:55:50,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:50,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:50,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:50,556 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:50,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:50,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:50,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362610552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362610557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:50,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:50,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:50,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:50,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362610558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362610559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e2aca5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:50,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:50,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:50,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:50,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a711e62ea5c7481ebe1f1d3efdd68af6 2023-04-24 18:55:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,579 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:50,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/1ea8879e58b44e5ea55e0057e7690fc5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5 2023-04-24 18:55:50,582 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:50,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:50,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:50,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:50,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:50,583 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:50,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,586 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5, entries=200, sequenceid=233, filesize=14.1 K 2023-04-24 18:55:50,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/35c2eddee62d42539e81c7206423eeae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae 2023-04-24 18:55:50,590 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:50,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:50,591 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:50,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,590 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,591 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,591 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:50,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae, entries=150, sequenceid=233, filesize=11.8 K 2023-04-24 18:55:50,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/a711e62ea5c7481ebe1f1d3efdd68af6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6 2023-04-24 18:55:50,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6, entries=150, sequenceid=233, filesize=11.8 K 2023-04-24 18:55:50,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 55e964c86d584c4edcab7a055c74c603 in 74ms, sequenceid=233, compaction requested=true 2023-04-24 18:55:50,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:50,602 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:50,602 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:50,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:50,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38942 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:50,605 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,605 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,605 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/9487a087659c4ce3b9dbf4506d60c60c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=38.0 K 2023-04-24 18:55:50,605 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7127a1e5e1644c4bbc8091a398eeb001, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=35.7 K 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9487a087659c4ce3b9dbf4506d60c60c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:50,605 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7127a1e5e1644c4bbc8091a398eeb001, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:50,606 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 77f25a29fad5426eb01181bac56a50c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682362549070 2023-04-24 18:55:50,606 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b827a5293b504b53aa27e407d1223083, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682362549070 2023-04-24 18:55:50,607 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1ea8879e58b44e5ea55e0057e7690fc5, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550198 2023-04-24 18:55:50,607 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 35c2eddee62d42539e81c7206423eeae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550213 2023-04-24 18:55:50,615 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:50,617 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:50,653 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/badec22280254364a58fdef70048c333 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/badec22280254364a58fdef70048c333 2023-04-24 18:55:50,664 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5ac7a879bb824a0cb54c9cd9985cf05b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ac7a879bb824a0cb54c9cd9985cf05b 2023-04-24 18:55:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:50,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:50,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:50,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 5ac7a879bb824a0cb54c9cd9985cf05b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:50,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362550602; duration=0sec 2023-04-24 18:55:50,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:50,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:50,676 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:50,676 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:50,676 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,676 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e2d475364fc44fa588cdb5dd606b0f60, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=35.7 K 2023-04-24 18:55:50,676 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e2d475364fc44fa588cdb5dd606b0f60, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682362549041 2023-04-24 18:55:50,677 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting be332a0c4ab8469dab94d0b18573aaa0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682362549070 2023-04-24 18:55:50,677 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a711e62ea5c7481ebe1f1d3efdd68af6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550213 2023-04-24 18:55:50,683 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into badec22280254364a58fdef70048c333(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:50,683 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:50,683 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362550602; duration=0sec 2023-04-24 18:55:50,683 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:50,691 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:50,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:50,693 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:50,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:50,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362610683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362610687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:50,700 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:50,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362610696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362610696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:50,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:50,708 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,708 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:50,708 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:50,708 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:50,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13cc1f2b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:50,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:50,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,714 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:50,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,714 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:50,716 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:50,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:50,718 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:50,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:50,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:50,718 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:50,718 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:50,720 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,720 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,720 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,721 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,722 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,725 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,725 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:50,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:50,726 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,726 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:50,726 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:50,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362610797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362610798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362610806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:50,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362610806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,826 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:50,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:50,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:50,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:50,830 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:50,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:50,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:50,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,831 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,831 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:50,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:50,834 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,834 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:50,834 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:50,834 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:50,835 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9c30fe[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:50,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:50,842 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:50,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,845 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:50,846 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:50,849 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:50,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:50,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:50,849 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:50,849 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:50,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,850 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,851 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,852 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,854 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,855 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:50,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:50,856 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:50,856 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:50,956 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:50,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:50,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:50,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:50,959 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,961 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,961 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:50,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:50,963 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,963 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:50,963 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:50,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:50,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:50,966 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a1e68d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:50,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:50,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,970 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:50,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,971 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:50,972 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:50,975 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:50,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:50,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:50,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:50,975 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:50,981 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:50,981 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,981 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:50,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:50,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:50,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:50,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:50,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:50,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:50,986 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:50,986 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:50,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362611002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362611002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362611009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362611011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,086 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,089 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,089 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,090 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,090 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,091 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,094 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,094 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,096 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,098 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,098 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b2e138[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,099 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,100 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,100 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,101 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,106 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,106 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/bb490f366074495f9c2fa722da2b41b7 2023-04-24 18:55:51,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,109 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,109 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,115 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,115 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,117 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/3736801002ea4f8db752ffc841c28438 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/3736801002ea4f8db752ffc841c28438 2023-04-24 18:55:51,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,124 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 3736801002ea4f8db752ffc841c28438(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:51,124 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:51,124 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362550602; duration=0sec 2023-04-24 18:55:51,124 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:51,216 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,219 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,219 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,220 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,220 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,222 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,222 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,224 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362611218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43748542[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,229 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,230 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,231 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,236 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,236 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,239 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,239 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,240 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,240 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,244 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,245 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,245 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362611307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362611308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362611315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362611315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,345 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,348 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,348 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,348 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,348 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,349 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,349 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,350 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,352 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,352 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,352 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,352 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,352 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,354 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e125612[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,358 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,359 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,360 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,360 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,371 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,371 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,371 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,371 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,471 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,475 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,475 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,476 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,476 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,477 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,477 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,478 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,478 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,478 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,478 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,479 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,479 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,479 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,479 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,479 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,480 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ac4ad3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,482 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,482 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,483 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,483 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,483 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,486 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,486 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,486 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,486 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,488 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,488 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,489 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,491 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,495 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,495 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,495 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,495 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,495 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5ec1094c635b46ed893eab409d67ed96 2023-04-24 18:55:51,595 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,598 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,599 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,600 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,600 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,601 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,602 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,602 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,603 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,603 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3230d27f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,609 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,609 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,614 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,614 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,615 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,615 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,615 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,621 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,621 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,621 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,721 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,724 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,725 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,726 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,726 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,727 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,728 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,728 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,728 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,728 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,728 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,730 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,730 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,732 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,732 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55891fee[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,734 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,734 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,734 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,735 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,741 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,743 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,743 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,745 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,750 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,751 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,751 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,751 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,751 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1682362611810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53044 deadline: 1682362611815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53080 deadline: 1682362611821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53074 deadline: 1682362611821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,851 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,855 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,855 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,856 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,858 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,858 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:51,861 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:51,862 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7987618b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:51,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:51,865 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,866 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:51,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:51,867 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:51,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:51,870 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:51,870 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:51,870 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:51,870 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:51,870 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:51,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,874 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:51,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,877 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:51,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:51,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:51,877 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:51,877 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:51,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/cb668cc5683e4941ab76adae66e1fe8c 2023-04-24 18:55:51,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/bb490f366074495f9c2fa722da2b41b7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7 2023-04-24 18:55:51,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7, entries=200, sequenceid=253, filesize=14.1 K 2023-04-24 18:55:51,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/5ec1094c635b46ed893eab409d67ed96 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96 2023-04-24 18:55:51,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96, entries=150, sequenceid=253, filesize=11.8 K 2023-04-24 18:55:51,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/cb668cc5683e4941ab76adae66e1fe8c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c 2023-04-24 18:55:51,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c, entries=150, sequenceid=253, filesize=11.8 K 2023-04-24 18:55:51,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 55e964c86d584c4edcab7a055c74c603 in 1285ms, sequenceid=253, compaction requested=false 2023-04-24 18:55:51,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:51,977 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:51,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:51,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:51,981 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:51,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:51,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:51,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,982 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:51,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:51,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:51,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:51,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:51,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:51,985 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:51,985 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:51,985 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:55:51,985 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:51,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:51,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:51,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:51,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:51,986 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:51,995 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f241f7fb78f54bf6926d1df0bc36be63 2023-04-24 18:55:52,009 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/4323314a580a47c480908df96c63b398 2023-04-24 18:55:52,422 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/9dad4bb7b81c4834b13884432c8abe18 2023-04-24 18:55:52,426 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f241f7fb78f54bf6926d1df0bc36be63 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63 2023-04-24 18:55:52,429 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63, entries=150, sequenceid=269, filesize=11.9 K 2023-04-24 18:55:52,430 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/4323314a580a47c480908df96c63b398 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398 2023-04-24 18:55:52,433 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398, entries=150, sequenceid=269, filesize=11.9 K 2023-04-24 18:55:52,434 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/9dad4bb7b81c4834b13884432c8abe18 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18 2023-04-24 18:55:52,437 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18, entries=150, sequenceid=269, filesize=11.9 K 2023-04-24 18:55:52,438 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 55e964c86d584c4edcab7a055c74c603 in 453ms, sequenceid=269, compaction requested=true 2023-04-24 18:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:52,440 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0a46ba[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:52,442 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:52,442 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:52,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:52,446 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:52,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:52,449 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:52,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:52,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:52,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:52,449 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:52,450 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,450 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,450 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,451 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,452 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,454 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,455 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,455 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:52,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,456 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:52,456 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:52,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,556 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:52,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:52,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:52,559 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:52,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:52,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:52,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,562 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:52,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:52,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:52,563 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:52,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:52,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:52,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:52,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:52,564 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:52,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:52,566 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,567 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,567 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1dce2e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:52,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:52,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:52,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:52,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:52,571 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:52,573 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:52,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:52,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:52,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:52,574 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:52,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,580 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,582 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,583 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,583 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:52,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:52,583 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:52,583 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:52,683 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:52,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:52,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:52,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:52,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:52,687 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:52,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:52,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:52,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:52,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:52,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:52,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:52,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:52,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:52,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:52,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f6c0ec[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:52,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,700 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:52,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:52,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:52,704 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:52,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:52,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:52,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:52,704 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:52,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:52,711 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,712 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:52,713 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:52,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,716 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:52,716 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,715 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:52,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:52,716 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:52,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:52,816 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:52,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:52,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:52,819 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:52,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:52,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:52,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:52,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:52,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:52,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:52,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:52,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:52,825 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:52,825 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:52,825 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-24 18:55:52,826 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:52,826 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:52,826 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:52,826 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:52,827 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:52,827 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:52,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:52,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362552846, max timestamp=1682362552864Num uniques 859; 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362552846, max timestamp=1682362552864Num uniques 859; 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362552846, max timestamp=1682362552864Num uniques 859; 2023-04-24 18:55:52,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:52,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362552894, max timestamp=1682362552898Num uniques 1018; 2023-04-24 18:55:52,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:55:52,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362552894, max timestamp=1682362552898Num uniques 1018; 2023-04-24 18:55:52,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682362552894, max timestamp=1682362552898Num uniques 1018; 2023-04-24 18:55:52,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:55:52,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 18:55:52,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362552928, max timestamp=1682362552937Num uniques 1027; 2023-04-24 18:55:52,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:55:52,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362552928, max timestamp=1682362552937Num uniques 1027; 2023-04-24 18:55:52,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:55:52,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362552928, max timestamp=1682362552937Num uniques 1027; 2023-04-24 18:55:52,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362552966, max timestamp=1682362552975Num uniques 1086; 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362552966, max timestamp=1682362552975Num uniques 1086; 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682362552966, max timestamp=1682362552975Num uniques 1086; 2023-04-24 18:55:52,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-24 18:55:53,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362613006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362613006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362613008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362613013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362613116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362613116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362613120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362613120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,235 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/56f1af9e6826436dab62dbc3b911bffc 2023-04-24 18:55:53,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362613235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,244 DEBUG [Thread-2956] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:53,250 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/a19f431fcbb044339369c9b5e96d299d 2023-04-24 18:55:53,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362613328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362613329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362613329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362613330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362613633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362613633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362613634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362613636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,659 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/d5454696216a4d59b2763e61db1193a3 2023-04-24 18:55:53,663 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/56f1af9e6826436dab62dbc3b911bffc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc 2023-04-24 18:55:53,668 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc, entries=250, sequenceid=279, filesize=16.7 K 2023-04-24 18:55:53,668 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/a19f431fcbb044339369c9b5e96d299d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d 2023-04-24 18:55:53,672 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d, entries=150, sequenceid=279, filesize=11.9 K 2023-04-24 18:55:53,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/d5454696216a4d59b2763e61db1193a3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3 2023-04-24 18:55:53,676 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3, entries=150, sequenceid=279, filesize=11.9 K 2023-04-24 18:55:53,677 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 55e964c86d584c4edcab7a055c74c603 in 852ms, sequenceid=279, compaction requested=true 2023-04-24 18:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:53,678 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653f76a0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:53,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:53,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,683 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:53,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:53,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:53,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:53,687 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:53,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:53,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:53,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:53,687 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:53,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,688 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:53,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:53,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:53,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:53,694 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:53,694 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:53,694 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:53,794 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:53,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:53,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:53,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:53,797 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:53,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:53,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:53,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:53,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,798 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:53,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:53,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:53,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:53,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:53,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:53,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:53,802 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=147.60 KB heapSize=412.92 KB 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:53,813 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/210f72d97355426bb35928a40591aea8 2023-04-24 18:55:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:54,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:54,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362614147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362614148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362614152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362614153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,245 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/6e514137a2bd4e94bbaef882eccb7198 2023-04-24 18:55:54,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362614254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362614256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362614260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362614260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,270 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ce45af2ac02f4b0f9d81f31899de17d5 2023-04-24 18:55:54,274 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/210f72d97355426bb35928a40591aea8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8 2023-04-24 18:55:54,277 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8, entries=150, sequenceid=367, filesize=11.9 K 2023-04-24 18:55:54,278 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/6e514137a2bd4e94bbaef882eccb7198 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198 2023-04-24 18:55:54,292 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198, entries=150, sequenceid=367, filesize=11.9 K 2023-04-24 18:55:54,293 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ce45af2ac02f4b0f9d81f31899de17d5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5 2023-04-24 18:55:54,296 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5, entries=150, sequenceid=367, filesize=11.9 K 2023-04-24 18:55:54,297 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~412.88 KB/422784, currentSize=33.54 KB/34350 for 55e964c86d584c4edcab7a055c74c603 in 495ms, sequenceid=367, compaction requested=true 2023-04-24 18:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:54,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:54,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:54,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:54,298 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395cd32b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:54,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:54,301 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,305 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:54,305 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:54,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:54,309 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:54,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:54,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:54,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:54,309 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:54,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:54,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,316 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,318 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:54,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:54,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:54,323 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:54,323 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:54,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:54,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,323 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:54,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:54,423 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:54,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:54,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:54,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:54,426 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:54,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:54,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:54,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:54,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:54,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:54,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,430 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:54,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:54,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:54,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:54,432 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 18:55:54,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:54,432 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:54,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:54,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:54,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:54,433 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:54,451 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/32e5d4b68a3c467ca8a39e012a83efbb 2023-04-24 18:55:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:54,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:54,480 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/d383a6fb1b9347d4a8f3515511719f2a 2023-04-24 18:55:54,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362554497, max timestamp=1682362554518Num uniques 859; 2023-04-24 18:55:54,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362554497, max timestamp=1682362554518Num uniques 859; 2023-04-24 18:55:54,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:54,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362554497, max timestamp=1682362554518Num uniques 859; 2023-04-24 18:55:54,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:54,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:54,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362614558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362554547, max timestamp=1682362554561Num uniques 1068; 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362554547, max timestamp=1682362554561Num uniques 1068; 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362554547, max timestamp=1682362554561Num uniques 1068; 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:54,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 18:55:54,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362614611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362614612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362614621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362614664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362614722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362614722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362614730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362614870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,891 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/56ec93719e404236bff94ffacc6bf1f8 2023-04-24 18:55:54,894 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/32e5d4b68a3c467ca8a39e012a83efbb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb 2023-04-24 18:55:54,897 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb, entries=100, sequenceid=375, filesize=9.5 K 2023-04-24 18:55:54,898 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/d383a6fb1b9347d4a8f3515511719f2a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a 2023-04-24 18:55:54,901 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a, entries=100, sequenceid=375, filesize=9.5 K 2023-04-24 18:55:54,902 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/56ec93719e404236bff94ffacc6bf1f8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8 2023-04-24 18:55:54,906 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8, entries=100, sequenceid=375, filesize=9.5 K 2023-04-24 18:55:54,907 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 55e964c86d584c4edcab7a055c74c603 in 475ms, sequenceid=375, compaction requested=true 2023-04-24 18:55:54,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:54,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:54,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:54,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:54,908 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:54,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:54,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d640690[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:54,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:54,912 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,917 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:54,917 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,917 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:54,919 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:54,919 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:54,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:54,922 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:54,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:54,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:54,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:54,922 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:54,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:54,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,923 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:54,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:54,924 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,925 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:54,927 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:54,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:54,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:54,927 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:54,927 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:54,928 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:54,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:54,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:54,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=154.31 KB heapSize=435.23 KB 2023-04-24 18:55:54,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362554597, max timestamp=1682362554933Num uniques 1127; 2023-04-24 18:55:54,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:54,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:54,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:54,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:54,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362554597, max timestamp=1682362554933Num uniques 1127; 2023-04-24 18:55:54,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682362554597, max timestamp=1682362554933Num uniques 1127; 2023-04-24 18:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 18:55:54,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:54,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:54,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:54,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/c8b486b10e6d49d9b0dd906540168200 2023-04-24 18:55:55,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362554975, max timestamp=1682362555007Num uniques 859; 2023-04-24 18:55:55,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:55,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362554975, max timestamp=1682362555007Num uniques 859; 2023-04-24 18:55:55,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682362554975, max timestamp=1682362555007Num uniques 859; 2023-04-24 18:55:55,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:55,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 18:55:55,028 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,031 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,034 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,034 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,036 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,037 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c045a70[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,048 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,050 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:55,053 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,053 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,053 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362555059, max timestamp=1682362555080Num uniques 1118; 2023-04-24 18:55:55,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 18:55:55,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362555059, max timestamp=1682362555080Num uniques 1118; 2023-04-24 18:55:55,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 18:55:55,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682362555059, max timestamp=1682362555080Num uniques 1118; 2023-04-24 18:55:55,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362615070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 18:55:55,153 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,156 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,158 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,159 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,159 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,159 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,159 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,159 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6210b926[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,163 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,165 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,167 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,167 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,170 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,170 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,170 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,172 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362555119, max timestamp=1682362555174Num uniques 1127; 2023-04-24 18:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362555119, max timestamp=1682362555174Num uniques 1127; 2023-04-24 18:55:55,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-24 18:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682362555119, max timestamp=1682362555174Num uniques 1127; 2023-04-24 18:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-24 18:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-24 18:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,179 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,180 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,180 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,180 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,180 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362555211, max timestamp=1682362555223Num uniques 1086; 2023-04-24 18:55:55,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:55:55,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362555211, max timestamp=1682362555223Num uniques 1086; 2023-04-24 18:55:55,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:55:55,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682362555211, max timestamp=1682362555223Num uniques 1086; 2023-04-24 18:55:55,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-24 18:55:55,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362555251, max timestamp=1682362555270Num uniques 1045; 2023-04-24 18:55:55,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:55:55,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362555251, max timestamp=1682362555270Num uniques 1045; 2023-04-24 18:55:55,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1682362555251, max timestamp=1682362555270Num uniques 1045; 2023-04-24 18:55:55,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:55:55,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=20380, suffix heap size=136648,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=445 2023-04-24 18:55:55,280 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,283 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,283 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,289 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0afbc4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,292 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,294 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,295 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,297 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,297 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,297 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,297 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,298 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,303 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,307 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,307 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:55,308 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,308 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362615321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362615331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362615332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362615333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/883f623b39304046883865a6c64e4b81 2023-04-24 18:55:55,408 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,411 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,412 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,414 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,414 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,414 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,414 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,414 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,415 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,416 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70173a73[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,420 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,421 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,423 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,423 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:55,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,428 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,428 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,429 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362615434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362615448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362615452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362615452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,529 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,532 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,532 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,537 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1e29db[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,544 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,548 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,548 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,554 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,557 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,557 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,557 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,557 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362615647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,657 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,660 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49207cb7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:55,667 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:55,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362615663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,670 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:55,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:55,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362615664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362615664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:55,674 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:55,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:55,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:55,674 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:55,674 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:55,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,676 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:55,677 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,678 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:55,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:55,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:55,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:55,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,684 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:55,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,685 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:55,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:55,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:55,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/892b3c579752457089499eccffc3d74d 2023-04-24 18:55:55,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/c8b486b10e6d49d9b0dd906540168200 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200 2023-04-24 18:55:55,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200, entries=150, sequenceid=431, filesize=11.9 K 2023-04-24 18:55:55,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/883f623b39304046883865a6c64e4b81 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81 2023-04-24 18:55:55,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81, entries=150, sequenceid=431, filesize=11.9 K 2023-04-24 18:55:55,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/892b3c579752457089499eccffc3d74d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d 2023-04-24 18:55:55,785 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:55,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:55,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:55,790 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:55,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:55,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:55,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d, entries=150, sequenceid=431, filesize=11.9 K 2023-04-24 18:55:55,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~84.42 KB/86448, currentSize=161.02 KB/164880 for 55e964c86d584c4edcab7a055c74c603 in 857ms, sequenceid=431, compaction requested=true 2023-04-24 18:55:55,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:55,791 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 18:55:55,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:55,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:55,791 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 18:55:55,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:55,791 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:55,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:55,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:55,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:55,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:55,794 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 90374 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 18:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:55,794 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:55,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,794 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:55,794 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:55,794 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/badec22280254364a58fdef70048c333, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=88.3 K 2023-04-24 18:55:55,794 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83104 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 18:55:55,794 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:55,794 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,794 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ac7a879bb824a0cb54c9cd9985cf05b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=81.2 K 2023-04-24 18:55:55,794 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=161.02 KB heapSize=437.86 KB 2023-04-24 18:55:55,795 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting badec22280254364a58fdef70048c333, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550213 2023-04-24 18:55:55,795 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5ac7a879bb824a0cb54c9cd9985cf05b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550213 2023-04-24 18:55:55,795 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:55,795 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting bb490f366074495f9c2fa722da2b41b7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682362550556 2023-04-24 18:55:55,795 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5ec1094c635b46ed893eab409d67ed96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682362550556 2023-04-24 18:55:55,795 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:55,796 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4323314a580a47c480908df96c63b398, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682362550680 2023-04-24 18:55:55,796 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f241f7fb78f54bf6926d1df0bc36be63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682362550680 2023-04-24 18:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:55,796 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a19f431fcbb044339369c9b5e96d299d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682362552822 2023-04-24 18:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:55,796 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 56f1af9e6826436dab62dbc3b911bffc, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682362552820 2023-04-24 18:55:55,798 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6e514137a2bd4e94bbaef882eccb7198, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682362553004 2023-04-24 18:55:55,798 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 210f72d97355426bb35928a40591aea8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682362553004 2023-04-24 18:55:55,798 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d383a6fb1b9347d4a8f3515511719f2a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682362554151 2023-04-24 18:55:55,798 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 32e5d4b68a3c467ca8a39e012a83efbb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682362554151 2023-04-24 18:55:55,799 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 883f623b39304046883865a6c64e4b81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:55,799 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c8b486b10e6d49d9b0dd906540168200, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:55,807 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/e432829b51224a1981ceb0a893e650df 2023-04-24 18:55:55,813 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#448 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:55,830 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#450 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:55,836 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/0120926596194c7685dde994fa72149a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0120926596194c7685dde994fa72149a 2023-04-24 18:55:55,842 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 0120926596194c7685dde994fa72149a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:55,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:55,842 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=9, startTime=1682362555791; duration=0sec 2023-04-24 18:55:55,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:55,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 18:55:55,850 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/025c8b2709cb42f7a84f5418676ab2bb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/025c8b2709cb42f7a84f5418676ab2bb 2023-04-24 18:55:55,852 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83104 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 18:55:55,853 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:55,853 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:55,853 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/3736801002ea4f8db752ffc841c28438, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=81.2 K 2023-04-24 18:55:55,853 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3736801002ea4f8db752ffc841c28438, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682362550213 2023-04-24 18:55:55,853 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cb668cc5683e4941ab76adae66e1fe8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682362550556 2023-04-24 18:55:55,854 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9dad4bb7b81c4834b13884432c8abe18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682362550680 2023-04-24 18:55:55,854 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 025c8b2709cb42f7a84f5418676ab2bb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:55,854 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:55,854 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=9, startTime=1682362555791; duration=0sec 2023-04-24 18:55:55,855 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:55,855 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d5454696216a4d59b2763e61db1193a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682362552822 2023-04-24 18:55:55,855 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting ce45af2ac02f4b0f9d81f31899de17d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682362553004 2023-04-24 18:55:55,855 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 56ec93719e404236bff94ffacc6bf1f8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682362554151 2023-04-24 18:55:55,856 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 892b3c579752457089499eccffc3d74d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:55,873 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#451 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:55,886 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/7b07f37fd72b4cb2b645de91c2fa13dc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7b07f37fd72b4cb2b645de91c2fa13dc 2023-04-24 18:55:55,896 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 7b07f37fd72b4cb2b645de91c2fa13dc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:55,896 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:55,896 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=9, startTime=1682362555791; duration=0sec 2023-04-24 18:55:55,896 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:55,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:55,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362615972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362615974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362615975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:55,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:55,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362615976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362616078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362616081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,238 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/a6c143c0039a4591843bf07b7c32ec37 2023-04-24 18:55:56,248 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/12aa72b2f4c24d2184ac4cf638a5420f 2023-04-24 18:55:56,251 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/e432829b51224a1981ceb0a893e650df as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df 2023-04-24 18:55:56,255 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df, entries=150, sequenceid=535, filesize=11.9 K 2023-04-24 18:55:56,255 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/a6c143c0039a4591843bf07b7c32ec37 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37 2023-04-24 18:55:56,261 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37, entries=150, sequenceid=535, filesize=11.9 K 2023-04-24 18:55:56,262 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/12aa72b2f4c24d2184ac4cf638a5420f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f 2023-04-24 18:55:56,265 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f, entries=150, sequenceid=535, filesize=11.9 K 2023-04-24 18:55:56,266 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~437.81 KB/448320, currentSize=26.84 KB/27480 for 55e964c86d584c4edcab7a055c74c603 in 472ms, sequenceid=535, compaction requested=false 2023-04-24 18:55:56,266 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:56,266 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:56,267 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:56,273 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231e6a8c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:56,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:56,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,276 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:56,276 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:56,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:56,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:56,281 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:56,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:56,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:56,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:56,281 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:56,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:56,290 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:55:56,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:56,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:56,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:56,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:56,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:56,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:56,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:56,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,291 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:56,291 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:56,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:56,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362556330, max timestamp=1682362556346Num uniques 859; 2023-04-24 18:55:56,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362556330, max timestamp=1682362556346Num uniques 859; 2023-04-24 18:55:56,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:56,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362556330, max timestamp=1682362556346Num uniques 859; 2023-04-24 18:55:56,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:56,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:55:56,392 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:56,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:56,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:56,395 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,395 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,396 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,397 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:56,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:56,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:56,398 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,398 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:56,398 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:56,398 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:56,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:56,400 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a93083e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:56,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:56,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,404 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:56,404 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,405 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:56,405 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:56,408 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:56,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:56,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:56,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:56,408 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:56,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,409 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,410 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:56,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:56,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,414 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,414 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,414 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:56,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362616404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362616416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362616481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362616484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,514 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:56,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:56,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:56,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:56,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:56,517 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:56,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:56,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:56,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,520 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,521 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,521 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:56,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:56,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:56,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:56,523 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:56,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362616517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:56,524 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:56,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@351ca446[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:56,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:56,526 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,527 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:56,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:56,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:56,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:56,530 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:56,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:56,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:56,530 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:56,531 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:56,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362616521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,533 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,533 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,533 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,534 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,538 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:56,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:56,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,542 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,543 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,543 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,543 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,543 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,542 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:56,543 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:56,643 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:56,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:56,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:56,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:56,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:56,647 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:56,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:56,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:56,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,647 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,649 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:56,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:56,651 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,651 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:56,651 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:56,651 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,657 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f13b396[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:56,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:56,659 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,660 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,660 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:56,660 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:56,660 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:56,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:56,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:56,664 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:56,664 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:56,664 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:56,664 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:56,664 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:56,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:56,666 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:56,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:56,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:56,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,672 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:56,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,672 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:56,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:56,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/cbd3637ef1d94b5ab64faa25941ffa5e 2023-04-24 18:55:56,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/7452abfddbfa44c4aa3a3c24b3108ace 2023-04-24 18:55:56,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362616724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:56,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362616732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/2bb216d2e50f4253884d23ace9fd870a 2023-04-24 18:55:56,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/cbd3637ef1d94b5ab64faa25941ffa5e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e 2023-04-24 18:55:56,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e, entries=250, sequenceid=548, filesize=16.7 K 2023-04-24 18:55:56,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/7452abfddbfa44c4aa3a3c24b3108ace as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace 2023-04-24 18:55:56,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace, entries=150, sequenceid=548, filesize=11.9 K 2023-04-24 18:55:56,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/2bb216d2e50f4253884d23ace9fd870a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a 2023-04-24 18:55:56,772 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:56,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:56,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a, entries=150, sequenceid=548, filesize=11.9 K 2023-04-24 18:55:56,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 55e964c86d584c4edcab7a055c74c603 in 488ms, sequenceid=548, compaction requested=true 2023-04-24 18:55:56,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:56,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:56,778 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:56,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:56,778 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:56,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:56,779 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:56,779 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:56,779 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:56,779 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:56,779 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,779 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,779 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/025c8b2709cb42f7a84f5418676ab2bb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=41.2 K 2023-04-24 18:55:56,779 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0120926596194c7685dde994fa72149a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.5 K 2023-04-24 18:55:56,780 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 025c8b2709cb42f7a84f5418676ab2bb, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:56,780 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0120926596194c7685dde994fa72149a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:56,780 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e432829b51224a1981ceb0a893e650df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1682362555329 2023-04-24 18:55:56,780 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a6c143c0039a4591843bf07b7c32ec37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1682362555329 2023-04-24 18:55:56,780 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cbd3637ef1d94b5ab64faa25941ffa5e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:56,781 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7452abfddbfa44c4aa3a3c24b3108ace, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:56,783 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:56,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:56,783 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:56,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,790 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,792 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:56,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:56,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:56,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:56,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:56,797 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:56,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:56,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:56,805 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:56,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:56,807 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:56,822 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/5d07f147d1364659bc2e21460e7a3ab2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5d07f147d1364659bc2e21460e7a3ab2 2023-04-24 18:55:56,826 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 5d07f147d1364659bc2e21460e7a3ab2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:56,826 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:56,826 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362556778; duration=0sec 2023-04-24 18:55:56,827 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:56,827 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:56,828 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:56,828 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:56,828 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:56,828 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7b07f37fd72b4cb2b645de91c2fa13dc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.5 K 2023-04-24 18:55:56,829 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7b07f37fd72b4cb2b645de91c2fa13dc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1682362554620 2023-04-24 18:55:56,829 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 12aa72b2f4c24d2184ac4cf638a5420f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1682362555329 2023-04-24 18:55:56,830 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2bb216d2e50f4253884d23ace9fd870a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:56,841 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:56,854 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/7f5db1787abb4e87bc6407139f0f2952 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7f5db1787abb4e87bc6407139f0f2952 2023-04-24 18:55:56,858 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 7f5db1787abb4e87bc6407139f0f2952(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:56,858 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:56,858 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362556778; duration=0sec 2023-04-24 18:55:56,859 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:57,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:57,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:57,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362617059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362617065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362617169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362617171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,214 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/55f6354d2e4e4501ad8111136b0b2674 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/55f6354d2e4e4501ad8111136b0b2674 2023-04-24 18:55:57,218 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 55f6354d2e4e4501ad8111136b0b2674(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:57,218 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:57,218 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362556778; duration=0sec 2023-04-24 18:55:57,219 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:57,233 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/b7d9964e46f94e55b0daf76d1018798a 2023-04-24 18:55:57,244 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/77a0f16b3efc445697bf0c32d1c3d155 2023-04-24 18:55:57,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53094 deadline: 1682362617267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,270 DEBUG [Thread-2956] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:55:57,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362617375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362617378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362617490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362617498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,659 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ab28d5b2e5d441aa8a01d2963d384be3 2023-04-24 18:55:57,663 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/b7d9964e46f94e55b0daf76d1018798a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a 2023-04-24 18:55:57,667 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a, entries=150, sequenceid=585, filesize=11.9 K 2023-04-24 18:55:57,667 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/77a0f16b3efc445697bf0c32d1c3d155 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155 2023-04-24 18:55:57,671 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155, entries=150, sequenceid=585, filesize=11.9 K 2023-04-24 18:55:57,671 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/ab28d5b2e5d441aa8a01d2963d384be3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3 2023-04-24 18:55:57,674 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3, entries=150, sequenceid=585, filesize=11.9 K 2023-04-24 18:55:57,679 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 55e964c86d584c4edcab7a055c74c603 in 874ms, sequenceid=585, compaction requested=false 2023-04-24 18:55:57,679 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:57,680 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:57,681 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:57,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:55:57,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:57,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f89fcf2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:57,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:57,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:57,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:57,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:57,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:57,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:57,685 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:57,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,688 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:57,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:57,692 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:57,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:57,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:57,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:57,692 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:57,693 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,693 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,693 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:57,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:57,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:57,698 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,698 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:57,698 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:57,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:57,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362557722, max timestamp=1682362557739Num uniques 859; 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362557722, max timestamp=1682362557739Num uniques 859; 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362557722, max timestamp=1682362557739Num uniques 859; 2023-04-24 18:55:57,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:55:57,798 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:57,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:57,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362617794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:57,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362617795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:57,802 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,802 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,803 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:57,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:57,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,804 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:57,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:57,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:57,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:57,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:57,805 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:57,808 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da9c3c5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:57,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:57,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,812 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:57,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:57,813 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:57,817 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:57,817 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:57,817 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:57,817 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:57,817 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:57,820 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,820 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,820 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:57,824 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:57,825 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,825 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:57,825 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:57,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:57,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362617902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:57,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362617902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,925 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:57,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:57,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:57,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:57,929 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:57,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:57,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:57,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:57,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,929 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,930 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:57,930 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:57,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,931 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:57,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:57,932 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:57,937 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,937 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc762a2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:57,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:57,939 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:57,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:57,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:57,948 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:57,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:57,948 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:57,949 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,949 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,949 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:57,950 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,956 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:57,958 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:57,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:57,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:57,959 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:57,959 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:57,959 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:57,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,059 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:58,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:58,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:58,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:58,062 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:58,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:58,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:58,063 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,063 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,068 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:58,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:58,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:58,069 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,070 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:58,070 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:58,070 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,071 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6faaef8f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:58,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:58,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:58,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,074 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:58,075 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:58,078 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:58,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:58,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:58,078 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:58,078 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:58,083 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,083 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,084 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,084 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:58,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,087 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,087 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,087 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:58,088 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:58,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/97bcd95508194e65b790fe629f4f42c5 2023-04-24 18:55:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362618107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362618108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,188 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:58,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:58,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:58,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:58,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:58,191 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:58,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:58,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:58,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,193 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,194 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,194 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:58,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:58,196 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,196 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:58,196 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:58,196 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:58,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,198 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f24f0e4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:58,202 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:58,202 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,204 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:58,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,204 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:58,206 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:58,209 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:58,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:58,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:58,209 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:58,209 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:58,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,211 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,214 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,216 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:58,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,216 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,216 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:58,216 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:58,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,316 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:58,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:58,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:58,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:58,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:58,320 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:58,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:58,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:58,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,320 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,322 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:58,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:58,324 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:58,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:58,325 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,326 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:58,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:58,328 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c80ad2e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:58,331 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:58,331 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,332 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:58,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:58,336 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:58,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:58,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:58,337 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:58,337 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:58,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,338 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,339 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,340 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:58,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:58,342 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,342 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:58,343 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:58,342 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,343 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,343 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362618417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362618418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,443 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:58,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:58,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:58,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:58,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:58,446 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:58,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:58,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:58,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,448 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:58,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:58,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:58,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:58,450 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:58,453 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,453 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,453 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,453 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a640e6e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:58,455 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:58,455 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:58,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,456 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,457 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:58,460 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:58,460 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:58,460 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:58,460 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:58,460 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:58,463 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,463 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:58,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,463 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:58,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,463 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,465 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,468 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:58,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:58,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:58,468 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,468 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:58,468 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:58,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/653d510c9fbb4792810aae656f23c94a 2023-04-24 18:55:58,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e365cdfd31a843058085cb31c8366cbc 2023-04-24 18:55:58,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/97bcd95508194e65b790fe629f4f42c5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5 2023-04-24 18:55:58,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5, entries=200, sequenceid=599, filesize=14.3 K 2023-04-24 18:55:58,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/653d510c9fbb4792810aae656f23c94a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a 2023-04-24 18:55:58,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a, entries=150, sequenceid=599, filesize=11.9 K 2023-04-24 18:55:58,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e365cdfd31a843058085cb31c8366cbc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc 2023-04-24 18:55:58,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc, entries=150, sequenceid=599, filesize=11.9 K 2023-04-24 18:55:58,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 55e964c86d584c4edcab7a055c74c603 in 859ms, sequenceid=599, compaction requested=true 2023-04-24 18:55:58,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:58,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:58,543 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:58,543 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:58,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:58,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:58,544 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:55:58,545 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,545 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,545 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5d07f147d1364659bc2e21460e7a3ab2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=38.9 K 2023-04-24 18:55:58,545 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/55f6354d2e4e4501ad8111136b0b2674, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.6 K 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5d07f147d1364659bc2e21460e7a3ab2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 55f6354d2e4e4501ad8111136b0b2674, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:58,545 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b7d9964e46f94e55b0daf76d1018798a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682362556380 2023-04-24 18:55:58,546 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 77a0f16b3efc445697bf0c32d1c3d155, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682362556380 2023-04-24 18:55:58,546 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 97bcd95508194e65b790fe629f4f42c5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:55:58,546 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 653d510c9fbb4792810aae656f23c94a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:55:58,551 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:58,557 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#466 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:58,568 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:58,569 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/991dedf048bc4d7b87c27b8b3553cff9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/991dedf048bc4d7b87c27b8b3553cff9 2023-04-24 18:55:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:58,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:58,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:58,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:58,572 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:58,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:58,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:58,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:58,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:58,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:58,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:58,575 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 991dedf048bc4d7b87c27b8b3553cff9(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:58,575 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:58,575 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362558543; duration=0sec 2023-04-24 18:55:58,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:58,575 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:58,575 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:58,575 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:58,576 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:58,576 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:55:58,577 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:55:58,578 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:55:58,578 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:58,578 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7f5db1787abb4e87bc6407139f0f2952, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.6 K 2023-04-24 18:55:58,578 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7f5db1787abb4e87bc6407139f0f2952, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682362555958 2023-04-24 18:55:58,578 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ab28d5b2e5d441aa8a01d2963d384be3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682362556380 2023-04-24 18:55:58,579 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e365cdfd31a843058085cb31c8366cbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:55:58,584 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:55:58,592 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/8a4ab74147d4448b8d142eeb1e5723a3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/8a4ab74147d4448b8d142eeb1e5723a3 2023-04-24 18:55:58,596 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 8a4ab74147d4448b8d142eeb1e5723a3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:58,596 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:58,596 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362558544; duration=0sec 2023-04-24 18:55:58,596 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:58,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:58,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:58,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362618945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:58,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362618946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:58,962 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/121b9b79e8bf4483abbba204d4075253 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/121b9b79e8bf4483abbba204d4075253 2023-04-24 18:55:58,967 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 121b9b79e8bf4483abbba204d4075253(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:55:58,967 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:58,967 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362558543; duration=0sec 2023-04-24 18:55:58,967 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:55:58,984 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f1a1b10e73b8498db69523e8936b9b32 2023-04-24 18:55:58,995 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/318b275b625f4bb2af29c91472d4fde5 2023-04-24 18:55:59,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362619052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362619052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362619258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362619258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,404 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e4198375d51041db8dc0601bb72b35e6 2023-04-24 18:55:59,407 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/f1a1b10e73b8498db69523e8936b9b32 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32 2023-04-24 18:55:59,411 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32, entries=150, sequenceid=635, filesize=11.9 K 2023-04-24 18:55:59,412 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/318b275b625f4bb2af29c91472d4fde5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5 2023-04-24 18:55:59,414 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5, entries=150, sequenceid=635, filesize=11.9 K 2023-04-24 18:55:59,415 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e4198375d51041db8dc0601bb72b35e6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6 2023-04-24 18:55:59,420 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6, entries=150, sequenceid=635, filesize=11.9 K 2023-04-24 18:55:59,420 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 55e964c86d584c4edcab7a055c74c603 in 844ms, sequenceid=635, compaction requested=false 2023-04-24 18:55:59,420 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:55:59,420 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:59,422 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120a4a3a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:59,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:59,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:59,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:59,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:59,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:59,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:59,432 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:59,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:59,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:59,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:59,433 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:59,435 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,435 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,435 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,435 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,441 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:59,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,441 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,441 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:59,441 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,442 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,442 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:55:59,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:55:59,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:55:59,541 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:59,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:59,547 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:59,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:59,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:59,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,549 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:59,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:59,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:59,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:59,555 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a812b4f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:59,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:59,557 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:59,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:59,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:59,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:59,562 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:59,565 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:59,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:59,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:59,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:59,566 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:59,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,573 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362619569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362619570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,576 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362619570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:59,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:59,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362619572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,578 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:59,578 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:59,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:59,678 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:59,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:59,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362619677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362619677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362619678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362619679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:59,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:59,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:59,682 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:59,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:59,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:59,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,687 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:59,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:59,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:59,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:59,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:59,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:59,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:59,690 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:59,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:59,692 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237108ec[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:59,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:59,694 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:59,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:59,698 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:59,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:59,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:59,702 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:59,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:59,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:59,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:59,702 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:59,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,704 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,705 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,709 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:59,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,709 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:59,709 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:59,810 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:59,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:59,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:59,813 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:59,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:59,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:59,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,813 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,816 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:59,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:59,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:59,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:59,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:59,817 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:59,817 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,817 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:59,818 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:59,818 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,819 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:59,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296c7592[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:59,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,822 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:59,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:59,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:59,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:59,823 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:59,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:59,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:59,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:59,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:59,826 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:59,826 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:59,834 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,834 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,834 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,834 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:59,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:59,838 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,838 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,839 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:59,839 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:59,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:59,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362619881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362619882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362619882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362619883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/d56db4ad6984438fba8b9c664fb0c01a 2023-04-24 18:55:59,939 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:55:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:55:59,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:55:59,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:55:59,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:55:59,942 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:55:59,942 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,942 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:55:59,942 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:55:59,942 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,942 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,944 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:55:59,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:55:59,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:55:59,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:55:59,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:55:59,945 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:55:59,947 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f7a11f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:55:59,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:55:59,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,950 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:55:59,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:55:59,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:55:59,950 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,950 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:55:59,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:55:59,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:55:59,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:55:59,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:55:59,954 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:55:59,955 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,955 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,955 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:55:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:55:59,955 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:55:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,957 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:55:59,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:55:59,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:55:59,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:55:59,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:55:59,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:55:59,962 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:55:59,962 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:55:59,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:55:59,963 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:55:59,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:55:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,063 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:00,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:00,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:00,066 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:00,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:00,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:00,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,066 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,068 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,068 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:00,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:00,069 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,069 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:00,070 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:00,070 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:00,071 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,071 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9f243d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:00,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:00,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,074 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:00,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,079 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:00,079 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:00,079 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:00,079 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:00,081 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,081 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,081 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,082 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,085 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:00,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,086 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,085 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:00,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:00,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,086 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:00,086 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:00,186 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:00,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:00,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:00,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:00,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:00,190 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:00,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:00,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:00,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362620186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362620186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362620188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362620188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,191 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:00,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:00,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:00,193 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:00,195 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,195 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718261ea[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:00,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:00,198 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,199 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:00,200 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,200 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:00,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:00,201 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,201 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:00,203 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:00,203 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:00,203 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:00,203 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:00,203 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:00,204 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,204 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,205 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,205 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,207 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,210 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,210 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,210 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,210 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:00,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:00,210 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,211 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,211 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:00,211 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:00,311 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:00,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:00,315 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:00,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:00,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:00,316 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,316 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,317 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:00,318 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:00,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:00,320 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,321 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:00,321 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:00,321 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:00,321 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:00,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b885aad[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:00,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:00,324 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,325 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:00,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:00,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,326 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:00,327 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/ea906f5fde574a4d9206b6b7e7cbca4f 2023-04-24 18:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:00,334 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:00,334 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:00,334 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:00,334 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:00,334 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:00,335 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,336 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,336 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,336 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:00,338 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/6b1db79dc9d74b9199e63949c49b0c50 2023-04-24 18:56:00,343 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,343 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:00,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,343 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:00,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:00,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:00,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:00,344 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:00,344 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,344 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:00,345 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/d56db4ad6984438fba8b9c664fb0c01a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a 2023-04-24 18:56:00,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a, entries=200, sequenceid=649, filesize=14.3 K 2023-04-24 18:56:00,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/ea906f5fde574a4d9206b6b7e7cbca4f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f 2023-04-24 18:56:00,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f, entries=150, sequenceid=649, filesize=11.9 K 2023-04-24 18:56:00,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/6b1db79dc9d74b9199e63949c49b0c50 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50 2023-04-24 18:56:00,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50, entries=150, sequenceid=649, filesize=11.9 K 2023-04-24 18:56:00,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 55e964c86d584c4edcab7a055c74c603 in 849ms, sequenceid=649, compaction requested=true 2023-04-24 18:56:00,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:00,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:00,359 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:00,359 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:00,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:00,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:00,360 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:00,360 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:56:00,361 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,361 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/121b9b79e8bf4483abbba204d4075253, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=39.0 K 2023-04-24 18:56:00,361 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:00,361 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:56:00,361 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,361 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/991dedf048bc4d7b87c27b8b3553cff9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.7 K 2023-04-24 18:56:00,361 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 121b9b79e8bf4483abbba204d4075253, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:56:00,362 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 991dedf048bc4d7b87c27b8b3553cff9, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:56:00,362 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f1a1b10e73b8498db69523e8936b9b32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1682362557785 2023-04-24 18:56:00,362 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 318b275b625f4bb2af29c91472d4fde5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1682362557785 2023-04-24 18:56:00,362 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d56db4ad6984438fba8b9c664fb0c01a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:00,363 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ea906f5fde574a4d9206b6b7e7cbca4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:00,370 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:00,370 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:00,391 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/3825bc99f87e4d0c82ccf936b90c6f45 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/3825bc99f87e4d0c82ccf936b90c6f45 2023-04-24 18:56:00,394 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 3825bc99f87e4d0c82ccf936b90c6f45(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:00,394 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:00,394 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362560359; duration=0sec 2023-04-24 18:56:00,394 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:00,394 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:00,395 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:00,395 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:56:00,395 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,395 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/8a4ab74147d4448b8d142eeb1e5723a3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.7 K 2023-04-24 18:56:00,396 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8a4ab74147d4448b8d142eeb1e5723a3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682362557053 2023-04-24 18:56:00,396 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e4198375d51041db8dc0601bb72b35e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1682362557785 2023-04-24 18:56:00,396 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6b1db79dc9d74b9199e63949c49b0c50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:00,400 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#476 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:00,444 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:00,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:00,447 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:00,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:00,447 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:00,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,448 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,450 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:00,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:00,450 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,451 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:00,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:00,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:00,453 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:56:00,453 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:00,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:00,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362620703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362620703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362620704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362620705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,790 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/9f8845b89f20443e9f58d114db63ae61 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9f8845b89f20443e9f58d114db63ae61 2023-04-24 18:56:00,794 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into 9f8845b89f20443e9f58d114db63ae61(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:00,794 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:00,794 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362560359; duration=0sec 2023-04-24 18:56:00,794 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:00,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362620810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362620810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,815 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/20a4a62a92ec4fa89494512643a11e0e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/20a4a62a92ec4fa89494512643a11e0e 2023-04-24 18:56:00,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362620812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,819 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 20a4a62a92ec4fa89494512643a11e0e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:00,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:00,819 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362620815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:00,819 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362560359; duration=0sec 2023-04-24 18:56:00,820 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:00,862 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/51edad9be15c4f2a96b167332f5c932d 2023-04-24 18:56:01,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362621014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362621016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362621017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362621020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,271 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/4b68e4b0383d42fd8cd973f4e007fb06 2023-04-24 18:56:01,280 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/7fe520ab99e24538889e6b823c4116e9 2023-04-24 18:56:01,283 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/51edad9be15c4f2a96b167332f5c932d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d 2023-04-24 18:56:01,287 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d, entries=150, sequenceid=670, filesize=11.9 K 2023-04-24 18:56:01,287 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/4b68e4b0383d42fd8cd973f4e007fb06 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06 2023-04-24 18:56:01,290 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06, entries=150, sequenceid=670, filesize=11.9 K 2023-04-24 18:56:01,290 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/7fe520ab99e24538889e6b823c4116e9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9 2023-04-24 18:56:01,294 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9, entries=150, sequenceid=670, filesize=11.9 K 2023-04-24 18:56:01,294 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 55e964c86d584c4edcab7a055c74c603 in 841ms, sequenceid=670, compaction requested=false 2023-04-24 18:56:01,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,294 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:01,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:01,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:01,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:01,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa3fb9a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:01,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:01,298 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,299 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:01,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:01,301 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:01,303 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:01,303 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:01,303 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:01,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:01,304 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:01,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,308 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,309 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,310 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,313 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:01,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,314 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:01,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,314 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:01,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:01,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,314 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:01,314 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:01,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:56:01,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/4422dba2e5a5410d91135948f0919984 2023-04-24 18:56:01,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/beb403b1b49f46a2a833b7a49e90252b 2023-04-24 18:56:01,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/6762ca7445664afb8626bbdb0f1f78d4 2023-04-24 18:56:01,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/4422dba2e5a5410d91135948f0919984 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984 2023-04-24 18:56:01,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984, entries=250, sequenceid=686, filesize=16.7 K 2023-04-24 18:56:01,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/beb403b1b49f46a2a833b7a49e90252b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b 2023-04-24 18:56:01,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b, entries=150, sequenceid=686, filesize=11.9 K 2023-04-24 18:56:01,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/6762ca7445664afb8626bbdb0f1f78d4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4 2023-04-24 18:56:01,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4, entries=150, sequenceid=686, filesize=11.9 K 2023-04-24 18:56:01,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=80.51 KB/82440 for 55e964c86d584c4edcab7a055c74c603 in 45ms, sequenceid=686, compaction requested=true 2023-04-24 18:56:01,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:01,370 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:01,370 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42528 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:56:01,371 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,371 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,371 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/3825bc99f87e4d0c82ccf936b90c6f45, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=41.5 K 2023-04-24 18:56:01,371 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9f8845b89f20443e9f58d114db63ae61, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.8 K 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 3825bc99f87e4d0c82ccf936b90c6f45, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:01,371 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9f8845b89f20443e9f58d114db63ae61, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:01,372 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 51edad9be15c4f2a96b167332f5c932d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682362559559 2023-04-24 18:56:01,372 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4b68e4b0383d42fd8cd973f4e007fb06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682362559559 2023-04-24 18:56:01,372 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4422dba2e5a5410d91135948f0919984, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560702 2023-04-24 18:56:01,372 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting beb403b1b49f46a2a833b7a49e90252b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560703 2023-04-24 18:56:01,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:01,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:56:01,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:01,378 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:01,380 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#485 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:01,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/b1a5dc13aa6f47778fff9b8b533eef2d 2023-04-24 18:56:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362621399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362621400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362621401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362621403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,414 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:01,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:01,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:01,417 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:01,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:01,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:01,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,418 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:01,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:01,422 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,422 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:01,422 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:01,422 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,424 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3e949f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:01,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:01,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,429 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:01,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:01,429 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:01,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,429 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:01,430 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:01,433 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:01,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:01,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:01,433 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:01,433 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:01,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,434 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,435 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:01,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:01,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,439 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:01,439 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:01,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362621504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362621508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362621508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362621511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,539 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:01,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:01,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:01,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:01,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:01,542 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:01,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:01,542 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:01,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,547 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:01,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:01,549 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,549 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:01,549 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:01,549 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,551 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e5131c3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:01,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:01,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,554 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,554 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:01,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:01,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:01,558 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:01,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:01,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:01,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:01,559 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:01,561 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,561 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,561 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,569 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,570 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:01,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:01,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:01,570 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:01,570 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:01,671 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:01,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:01,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:01,675 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:01,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:01,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:01,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,675 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,679 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,679 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:01,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:01,681 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,681 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:01,681 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:01,681 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:01,684 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,684 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57755b96[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:01,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:01,687 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,689 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:01,689 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:01,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:01,690 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:01,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,690 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:01,693 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:01,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:01,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:01,693 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:01,693 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:01,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,702 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:01,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:01,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,703 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:01,703 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:01,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:01,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,703 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53080 deadline: 1682362621708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1682362621713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53074 deadline: 1682362621714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:01,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53044 deadline: 1682362621715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,792 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6 2023-04-24 18:56:01,794 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/eb1c8b5c6fc84b51a4c73e8a285fc810 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/eb1c8b5c6fc84b51a4c73e8a285fc810 2023-04-24 18:56:01,797 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 66b9fbdfa9f84fcaa8324c9ab95c0ba6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:01,798 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,798 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362561370; duration=0sec 2023-04-24 18:56:01,798 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:01,798 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:01,799 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into eb1c8b5c6fc84b51a4c73e8a285fc810(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:01,799 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,799 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362561370; duration=0sec 2023-04-24 18:56:01,799 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:01,799 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:01,799 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:56:01,799 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,799 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/20a4a62a92ec4fa89494512643a11e0e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.8 K 2023-04-24 18:56:01,800 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 20a4a62a92ec4fa89494512643a11e0e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682362558939 2023-04-24 18:56:01,800 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 7fe520ab99e24538889e6b823c4116e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682362559559 2023-04-24 18:56:01,801 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 6762ca7445664afb8626bbdb0f1f78d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560703 2023-04-24 18:56:01,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/db3704e48d7449fb8e7452cadfef60d9 2023-04-24 18:56:01,803 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:01,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:01,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:01,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:01,807 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:01,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:01,807 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:01,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,808 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:01,809 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,809 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,811 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:01,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. started... 2023-04-24 18:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. as already flushing 2023-04-24 18:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:01,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:01,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/c7ed0477a2f744d699de8c58805caab5 2023-04-24 18:56:01,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:01,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fa05c9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:01,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:01,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,828 DEBUG [Thread-2967] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:59583 2023-04-24 18:56:01,828 DEBUG [Thread-2967] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:01,829 DEBUG [Thread-2971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:59583 2023-04-24 18:56:01,829 DEBUG [Thread-2975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:59583 2023-04-24 18:56:01,829 DEBUG [Thread-2971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:01,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:01,829 DEBUG [Thread-2975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/b1a5dc13aa6f47778fff9b8b533eef2d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d 2023-04-24 18:56:01,831 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/0c0700de45c648999e5a179a74d25ac2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/0c0700de45c648999e5a179a74d25ac2 2023-04-24 18:56:01,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,833 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:01,833 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d, entries=250, sequenceid=703, filesize=16.7 K 2023-04-24 18:56:01,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/db3704e48d7449fb8e7452cadfef60d9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9 2023-04-24 18:56:01,836 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 0c0700de45c648999e5a179a74d25ac2(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:01,836 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,836 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362561370; duration=0sec 2023-04-24 18:56:01,836 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:01,837 DEBUG [Thread-2969] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:59583 2023-04-24 18:56:01,837 DEBUG [Thread-2969] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:01,837 DEBUG [Thread-2973] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:59583 2023-04-24 18:56:01,837 DEBUG [Thread-2973] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:01,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9, entries=150, sequenceid=703, filesize=11.9 K 2023-04-24 18:56:01,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/c7ed0477a2f744d699de8c58805caab5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5 2023-04-24 18:56:01,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5, entries=150, sequenceid=703, filesize=11.9 K 2023-04-24 18:56:01,842 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 55e964c86d584c4edcab7a055c74c603 in 468ms, sequenceid=703, compaction requested=false 2023-04-24 18:56:01,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:01,937 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:01,937 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:01,937 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:01,937 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:01,937 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:01,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:01,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,940 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:01,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:01,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:01,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:01,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:01,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:01,943 DEBUG [Thread-2966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,943 DEBUG [Thread-2966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:01,943 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:01,944 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:02,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:02,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-24 18:56:02,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:56:02,016 DEBUG [Thread-2960] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:59583 2023-04-24 18:56:02,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:02,016 DEBUG [Thread-2960] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:02,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:56:02,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:02,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:56:02,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:02,021 DEBUG [Thread-2964] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:59583 2023-04-24 18:56:02,021 DEBUG [Thread-2964] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:02,021 DEBUG [Thread-2958] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:59583 2023-04-24 18:56:02,022 DEBUG [Thread-2958] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:02,023 DEBUG [Thread-2962] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:59583 2023-04-24 18:56:02,023 DEBUG [Thread-2962] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:02,044 DEBUG [Thread-2966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:02,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:02,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/5169e93d8c814d668c7862104b433df1 2023-04-24 18:56:02,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/1059dd11c69d48e2baeed3e7fed90736 2023-04-24 18:56:03,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e7d05faffd624062bcd19c1dcb1316b3 2023-04-24 18:56:03,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/5169e93d8c814d668c7862104b433df1 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1 2023-04-24 18:56:03,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1, entries=150, sequenceid=722, filesize=11.9 K 2023-04-24 18:56:03,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/1059dd11c69d48e2baeed3e7fed90736 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736 2023-04-24 18:56:03,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736, entries=150, sequenceid=722, filesize=11.9 K 2023-04-24 18:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/e7d05faffd624062bcd19c1dcb1316b3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3 2023-04-24 18:56:03,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3, entries=150, sequenceid=722, filesize=11.9 K 2023-04-24 18:56:03,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=20.13 KB/20610 for 55e964c86d584c4edcab7a055c74c603 in 1239ms, sequenceid=722, compaction requested=true 2023-04-24 18:56:03,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:03,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:03,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:03,255 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:03,255 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:03,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:03,256 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:03,256 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/A is initiating minor compaction (all files) 2023-04-24 18:56:03,256 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:03,256 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/A in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:03,256 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/B is initiating minor compaction (all files) 2023-04-24 18:56:03,257 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=41.6 K 2023-04-24 18:56:03,257 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/B in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:03,257 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/eb1c8b5c6fc84b51a4c73e8a285fc810, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.9 K 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 66b9fbdfa9f84fcaa8324c9ab95c0ba6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560703 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting eb1c8b5c6fc84b51a4c73e8a285fc810, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560703 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting b1a5dc13aa6f47778fff9b8b533eef2d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1682362561332 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting db3704e48d7449fb8e7452cadfef60d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1682362561332 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5169e93d8c814d668c7862104b433df1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1682362561394 2023-04-24 18:56:03,257 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1059dd11c69d48e2baeed3e7fed90736, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1682362561394 2023-04-24 18:56:03,263 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#B#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:03,264 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#A#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:03,670 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/e046020999bd495098aa824ee7e9698c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e046020999bd495098aa824ee7e9698c 2023-04-24 18:56:03,671 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/88c7ccc45c114c9396ff6d258356f79f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/88c7ccc45c114c9396ff6d258356f79f 2023-04-24 18:56:03,674 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/A of 55e964c86d584c4edcab7a055c74c603 into 88c7ccc45c114c9396ff6d258356f79f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:03,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/B of 55e964c86d584c4edcab7a055c74c603 into e046020999bd495098aa824ee7e9698c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:03,674 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:03,674 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:03,674 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/A, priority=13, startTime=1682362563255; duration=0sec 2023-04-24 18:56:03,674 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/B, priority=13, startTime=1682362563255; duration=0sec 2023-04-24 18:56:03,674 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:03,675 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:03,675 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:03,675 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:03,675 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 55e964c86d584c4edcab7a055c74c603/C is initiating minor compaction (all files) 2023-04-24 18:56:03,675 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 55e964c86d584c4edcab7a055c74c603/C in TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:03,676 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/0c0700de45c648999e5a179a74d25ac2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp, totalSize=36.9 K 2023-04-24 18:56:03,676 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 0c0700de45c648999e5a179a74d25ac2, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1682362560703 2023-04-24 18:56:03,676 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c7ed0477a2f744d699de8c58805caab5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1682362561332 2023-04-24 18:56:03,676 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e7d05faffd624062bcd19c1dcb1316b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1682362561394 2023-04-24 18:56:03,681 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 55e964c86d584c4edcab7a055c74c603#C#compaction#494 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:04,089 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/16ec637afd424e1893237992b9d968ab as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/16ec637afd424e1893237992b9d968ab 2023-04-24 18:56:04,092 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 55e964c86d584c4edcab7a055c74c603/C of 55e964c86d584c4edcab7a055c74c603 into 16ec637afd424e1893237992b9d968ab(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:04,092 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:04,092 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603., storeName=55e964c86d584c4edcab7a055c74c603/C, priority=13, startTime=1682362563255; duration=0sec 2023-04-24 18:56:04,092 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:07,287 DEBUG [Thread-2956] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:59583 2023-04-24 18:56:07,287 DEBUG [Thread-2956] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 168 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1926 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5778 rows 2023-04-24 18:56:07,287 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1947 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5841 rows 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1938 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5814 rows 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1939 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5817 rows 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(402): scanned 1948 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(403): verified 5844 rows 2023-04-24 18:56:07,288 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:56:07,288 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:59583 2023-04-24 18:56:07,288 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:07,289 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:56:07,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:56:07,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:07,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:07,294 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362567294"}]},"ts":"1682362567294"} 2023-04-24 18:56:07,295 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:56:07,296 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:56:07,297 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, UNASSIGN}] 2023-04-24 18:56:07,298 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, UNASSIGN 2023-04-24 18:56:07,298 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=55e964c86d584c4edcab7a055c74c603, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:07,299 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362567298"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362567298"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362567298"}]},"ts":"1682362567298"} 2023-04-24 18:56:07,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:56:07,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:07,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:07,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 55e964c86d584c4edcab7a055c74c603, disabling compactions & flushes 2023-04-24 18:56:07,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:07,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:07,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. after waiting 0 ms 2023-04-24 18:56:07,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:07,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 55e964c86d584c4edcab7a055c74c603 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=A 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=B 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 55e964c86d584c4edcab7a055c74c603, store=C 2023-04-24 18:56:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:07,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:07,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/fa4a97e1a9e54a2bb0ec8735029e7730 2023-04-24 18:56:07,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:08,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/501d0a0ef9914c37acee026fc4d719f8 2023-04-24 18:56:08,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:08,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/c1e03e7e44b44a5295d721bfaf58a551 2023-04-24 18:56:08,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/A/fa4a97e1a9e54a2bb0ec8735029e7730 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/fa4a97e1a9e54a2bb0ec8735029e7730 2023-04-24 18:56:08,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/fa4a97e1a9e54a2bb0ec8735029e7730, entries=50, sequenceid=732, filesize=7.1 K 2023-04-24 18:56:08,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/B/501d0a0ef9914c37acee026fc4d719f8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/501d0a0ef9914c37acee026fc4d719f8 2023-04-24 18:56:08,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/501d0a0ef9914c37acee026fc4d719f8, entries=50, sequenceid=732, filesize=7.1 K 2023-04-24 18:56:08,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/.tmp/C/c1e03e7e44b44a5295d721bfaf58a551 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c1e03e7e44b44a5295d721bfaf58a551 2023-04-24 18:56:08,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c1e03e7e44b44a5295d721bfaf58a551, entries=50, sequenceid=732, filesize=7.1 K 2023-04-24 18:56:08,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 55e964c86d584c4edcab7a055c74c603 in 1242ms, sequenceid=732, compaction requested=false 2023-04-24 18:56:08,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/360a3bd576084c1285a6020d8e75d10d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b37e5bce274248739f8e095a88c9745d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/9487a087659c4ce3b9dbf4506d60c60c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/badec22280254364a58fdef70048c333, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/025c8b2709cb42f7a84f5418676ab2bb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5d07f147d1364659bc2e21460e7a3ab2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/121b9b79e8bf4483abbba204d4075253, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/3825bc99f87e4d0c82ccf936b90c6f45, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1] to archive 2023-04-24 18:56:08,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:08,719 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/2a2c42f266414fe3b0ea1fd669c6539c 2023-04-24 18:56:08,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/092d62c1f96248efade4a9e5a59b3c28 2023-04-24 18:56:08,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/0383718c919140c6b69a13f3fc2caa71 2023-04-24 18:56:08,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210c2868b0604595b3f7975830faa2c4 2023-04-24 18:56:08,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/360a3bd576084c1285a6020d8e75d10d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/360a3bd576084c1285a6020d8e75d10d 2023-04-24 18:56:08,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e68bbaabb307411a9714504a0d31cfc2 2023-04-24 18:56:08,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f3ae041b9dfa49c49cdc2af8cf9f7d13 2023-04-24 18:56:08,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c4acd94010894c5fb9209eaf201fb01f 2023-04-24 18:56:08,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/692a1b5c3c0e4a42b7a8fb995724b19b 2023-04-24 18:56:08,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/314eb857867546cb9ad6ed73dc3a1471 2023-04-24 18:56:08,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b37e5bce274248739f8e095a88c9745d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b37e5bce274248739f8e095a88c9745d 2023-04-24 18:56:08,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/19c1db999cfe4bdfae1af08846db7ff3 2023-04-24 18:56:08,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d278520ca88f44c792cf99bee93cb0f0 2023-04-24 18:56:08,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/9487a087659c4ce3b9dbf4506d60c60c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/9487a087659c4ce3b9dbf4506d60c60c 2023-04-24 18:56:08,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/77f25a29fad5426eb01181bac56a50c2 2023-04-24 18:56:08,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/1ea8879e58b44e5ea55e0057e7690fc5 2023-04-24 18:56:08,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/badec22280254364a58fdef70048c333 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/badec22280254364a58fdef70048c333 2023-04-24 18:56:08,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/bb490f366074495f9c2fa722da2b41b7 2023-04-24 18:56:08,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f241f7fb78f54bf6926d1df0bc36be63 2023-04-24 18:56:08,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/56f1af9e6826436dab62dbc3b911bffc 2023-04-24 18:56:08,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/210f72d97355426bb35928a40591aea8 2023-04-24 18:56:08,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/32e5d4b68a3c467ca8a39e012a83efbb 2023-04-24 18:56:08,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/025c8b2709cb42f7a84f5418676ab2bb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/025c8b2709cb42f7a84f5418676ab2bb 2023-04-24 18:56:08,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/c8b486b10e6d49d9b0dd906540168200 2023-04-24 18:56:08,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/e432829b51224a1981ceb0a893e650df 2023-04-24 18:56:08,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/cbd3637ef1d94b5ab64faa25941ffa5e 2023-04-24 18:56:08,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5d07f147d1364659bc2e21460e7a3ab2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5d07f147d1364659bc2e21460e7a3ab2 2023-04-24 18:56:08,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b7d9964e46f94e55b0daf76d1018798a 2023-04-24 18:56:08,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/97bcd95508194e65b790fe629f4f42c5 2023-04-24 18:56:08,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/121b9b79e8bf4483abbba204d4075253 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/121b9b79e8bf4483abbba204d4075253 2023-04-24 18:56:08,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/f1a1b10e73b8498db69523e8936b9b32 2023-04-24 18:56:08,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/d56db4ad6984438fba8b9c664fb0c01a 2023-04-24 18:56:08,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/3825bc99f87e4d0c82ccf936b90c6f45 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/3825bc99f87e4d0c82ccf936b90c6f45 2023-04-24 18:56:08,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/51edad9be15c4f2a96b167332f5c932d 2023-04-24 18:56:08,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/4422dba2e5a5410d91135948f0919984 2023-04-24 18:56:08,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/66b9fbdfa9f84fcaa8324c9ab95c0ba6 2023-04-24 18:56:08,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/b1a5dc13aa6f47778fff9b8b533eef2d 2023-04-24 18:56:08,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/5169e93d8c814d668c7862104b433df1 2023-04-24 18:56:08,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5345ee885edc4ff58a6812c5caf14f48, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b2fb23dcce19424b97ca2f144dc16971, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7127a1e5e1644c4bbc8091a398eeb001, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ac7a879bb824a0cb54c9cd9985cf05b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0120926596194c7685dde994fa72149a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/55f6354d2e4e4501ad8111136b0b2674, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/991dedf048bc4d7b87c27b8b3553cff9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9f8845b89f20443e9f58d114db63ae61, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/eb1c8b5c6fc84b51a4c73e8a285fc810, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736] to archive 2023-04-24 18:56:08,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:08,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9ecf0f054eae4b2f9fb3eeb94ba6ef80 2023-04-24 18:56:08,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/efaa7857960a48e09bea672bbff37e4a 2023-04-24 18:56:08,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f73e2848f088491881c6be3f74786807 2023-04-24 18:56:08,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5345ee885edc4ff58a6812c5caf14f48 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5345ee885edc4ff58a6812c5caf14f48 2023-04-24 18:56:08,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9c83c99e3ca9499d8125279c428bdbc4 2023-04-24 18:56:08,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b681adfee4594dfb894ff215e4da4995 2023-04-24 18:56:08,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/843c09208ffd4f7590107be5dc21e05e 2023-04-24 18:56:08,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/17ce62562d6a49f19736503530080635 2023-04-24 18:56:08,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/f89555fd3af54ca59bcdde10a83eb84c 2023-04-24 18:56:08,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b2fb23dcce19424b97ca2f144dc16971 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b2fb23dcce19424b97ca2f144dc16971 2023-04-24 18:56:08,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0bf3a82d736c4b57883f6dbda9d1f352 2023-04-24 18:56:08,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5f51c559be964197b1cff3e3025c1ee1 2023-04-24 18:56:08,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7127a1e5e1644c4bbc8091a398eeb001 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7127a1e5e1644c4bbc8091a398eeb001 2023-04-24 18:56:08,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e417bff71e8f4903856d3046d9428e47 2023-04-24 18:56:08,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/b827a5293b504b53aa27e407d1223083 2023-04-24 18:56:08,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ac7a879bb824a0cb54c9cd9985cf05b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ac7a879bb824a0cb54c9cd9985cf05b 2023-04-24 18:56:08,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/35c2eddee62d42539e81c7206423eeae 2023-04-24 18:56:08,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/5ec1094c635b46ed893eab409d67ed96 2023-04-24 18:56:08,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4323314a580a47c480908df96c63b398 2023-04-24 18:56:08,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a19f431fcbb044339369c9b5e96d299d 2023-04-24 18:56:08,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/6e514137a2bd4e94bbaef882eccb7198 2023-04-24 18:56:08,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/d383a6fb1b9347d4a8f3515511719f2a 2023-04-24 18:56:08,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0120926596194c7685dde994fa72149a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/0120926596194c7685dde994fa72149a 2023-04-24 18:56:08,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/883f623b39304046883865a6c64e4b81 2023-04-24 18:56:08,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/a6c143c0039a4591843bf07b7c32ec37 2023-04-24 18:56:08,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/55f6354d2e4e4501ad8111136b0b2674 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/55f6354d2e4e4501ad8111136b0b2674 2023-04-24 18:56:08,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/7452abfddbfa44c4aa3a3c24b3108ace 2023-04-24 18:56:08,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/77a0f16b3efc445697bf0c32d1c3d155 2023-04-24 18:56:08,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/991dedf048bc4d7b87c27b8b3553cff9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/991dedf048bc4d7b87c27b8b3553cff9 2023-04-24 18:56:08,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/653d510c9fbb4792810aae656f23c94a 2023-04-24 18:56:08,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/318b275b625f4bb2af29c91472d4fde5 2023-04-24 18:56:08,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9f8845b89f20443e9f58d114db63ae61 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/9f8845b89f20443e9f58d114db63ae61 2023-04-24 18:56:08,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/ea906f5fde574a4d9206b6b7e7cbca4f 2023-04-24 18:56:08,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/4b68e4b0383d42fd8cd973f4e007fb06 2023-04-24 18:56:08,787 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/eb1c8b5c6fc84b51a4c73e8a285fc810 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/eb1c8b5c6fc84b51a4c73e8a285fc810 2023-04-24 18:56:08,788 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/beb403b1b49f46a2a833b7a49e90252b 2023-04-24 18:56:08,789 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/db3704e48d7449fb8e7452cadfef60d9 2023-04-24 18:56:08,790 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/1059dd11c69d48e2baeed3e7fed90736 2023-04-24 18:56:08,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7a73927c56074037948bddd509845b2b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/f3b9e12e75724adca3538e065e9a9a23, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e2d475364fc44fa588cdb5dd606b0f60, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/3736801002ea4f8db752ffc841c28438, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7b07f37fd72b4cb2b645de91c2fa13dc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7f5db1787abb4e87bc6407139f0f2952, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/8a4ab74147d4448b8d142eeb1e5723a3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/20a4a62a92ec4fa89494512643a11e0e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/0c0700de45c648999e5a179a74d25ac2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3] to archive 2023-04-24 18:56:08,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:08,795 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d65103e22b0d4dbf884736942d71ffb3 2023-04-24 18:56:08,796 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a9d51e396d0e4ae6bcef57a7848b3b40 2023-04-24 18:56:08,797 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/76f51b0b57e7432280b5dcf3fa3b8539 2023-04-24 18:56:08,797 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7a73927c56074037948bddd509845b2b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7a73927c56074037948bddd509845b2b 2023-04-24 18:56:08,798 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/53b5c6ff78d34ffb9632a1649605c529 2023-04-24 18:56:08,799 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a242746ac46a4ba69f7f26ba81ede1c3 2023-04-24 18:56:08,800 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cc6ef7a27ed84609b16e153fdc31650b 2023-04-24 18:56:08,801 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/fa0f9bdc057943469784f5f0c021b7f1 2023-04-24 18:56:08,802 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ebf593451b3f4fc4928ebfbf3e36d83d 2023-04-24 18:56:08,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/f3b9e12e75724adca3538e065e9a9a23 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/f3b9e12e75724adca3538e065e9a9a23 2023-04-24 18:56:08,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/143a91236c3a4b03b1a2f081dcd5901b 2023-04-24 18:56:08,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/1756e5678f1d439b9b1a411042bdb014 2023-04-24 18:56:08,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e2d475364fc44fa588cdb5dd606b0f60 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e2d475364fc44fa588cdb5dd606b0f60 2023-04-24 18:56:08,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/178150e6bf854af2b22cd29e56d47267 2023-04-24 18:56:08,811 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/be332a0c4ab8469dab94d0b18573aaa0 2023-04-24 18:56:08,812 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/3736801002ea4f8db752ffc841c28438 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/3736801002ea4f8db752ffc841c28438 2023-04-24 18:56:08,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/a711e62ea5c7481ebe1f1d3efdd68af6 2023-04-24 18:56:08,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/cb668cc5683e4941ab76adae66e1fe8c 2023-04-24 18:56:08,814 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/9dad4bb7b81c4834b13884432c8abe18 2023-04-24 18:56:08,815 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/d5454696216a4d59b2763e61db1193a3 2023-04-24 18:56:08,815 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ce45af2ac02f4b0f9d81f31899de17d5 2023-04-24 18:56:08,816 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/56ec93719e404236bff94ffacc6bf1f8 2023-04-24 18:56:08,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7b07f37fd72b4cb2b645de91c2fa13dc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7b07f37fd72b4cb2b645de91c2fa13dc 2023-04-24 18:56:08,818 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/892b3c579752457089499eccffc3d74d 2023-04-24 18:56:08,819 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/12aa72b2f4c24d2184ac4cf638a5420f 2023-04-24 18:56:08,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7f5db1787abb4e87bc6407139f0f2952 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7f5db1787abb4e87bc6407139f0f2952 2023-04-24 18:56:08,821 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/2bb216d2e50f4253884d23ace9fd870a 2023-04-24 18:56:08,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/ab28d5b2e5d441aa8a01d2963d384be3 2023-04-24 18:56:08,823 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/8a4ab74147d4448b8d142eeb1e5723a3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/8a4ab74147d4448b8d142eeb1e5723a3 2023-04-24 18:56:08,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e365cdfd31a843058085cb31c8366cbc 2023-04-24 18:56:08,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e4198375d51041db8dc0601bb72b35e6 2023-04-24 18:56:08,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/20a4a62a92ec4fa89494512643a11e0e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/20a4a62a92ec4fa89494512643a11e0e 2023-04-24 18:56:08,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6b1db79dc9d74b9199e63949c49b0c50 2023-04-24 18:56:08,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/7fe520ab99e24538889e6b823c4116e9 2023-04-24 18:56:08,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/0c0700de45c648999e5a179a74d25ac2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/0c0700de45c648999e5a179a74d25ac2 2023-04-24 18:56:08,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/6762ca7445664afb8626bbdb0f1f78d4 2023-04-24 18:56:08,831 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c7ed0477a2f744d699de8c58805caab5 2023-04-24 18:56:08,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/e7d05faffd624062bcd19c1dcb1316b3 2023-04-24 18:56:08,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/recovered.edits/735.seqid, newMaxSeqId=735, maxSeqId=1 2023-04-24 18:56:08,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603. 2023-04-24 18:56:08,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 55e964c86d584c4edcab7a055c74c603: 2023-04-24 18:56:08,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:08,839 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=55e964c86d584c4edcab7a055c74c603, regionState=CLOSED 2023-04-24 18:56:08,839 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362568839"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362568839"}]},"ts":"1682362568839"} 2023-04-24 18:56:08,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-24 18:56:08,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 55e964c86d584c4edcab7a055c74c603, server=jenkins-hbase4.apache.org,37593,1682362415680 in 1.5410 sec 2023-04-24 18:56:08,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-24 18:56:08,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=55e964c86d584c4edcab7a055c74c603, UNASSIGN in 1.5440 sec 2023-04-24 18:56:08,843 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362568843"}]},"ts":"1682362568843"} 2023-04-24 18:56:08,843 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:56:08,846 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:56:08,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5560 sec 2023-04-24 18:56:09,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 18:56:09,398 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-24 18:56:09,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:56:09,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,401 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-24 18:56:09,402 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,407 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:09,409 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/recovered.edits] 2023-04-24 18:56:09,411 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/88c7ccc45c114c9396ff6d258356f79f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/88c7ccc45c114c9396ff6d258356f79f 2023-04-24 18:56:09,412 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/fa4a97e1a9e54a2bb0ec8735029e7730 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/A/fa4a97e1a9e54a2bb0ec8735029e7730 2023-04-24 18:56:09,414 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/501d0a0ef9914c37acee026fc4d719f8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/501d0a0ef9914c37acee026fc4d719f8 2023-04-24 18:56:09,415 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e046020999bd495098aa824ee7e9698c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/B/e046020999bd495098aa824ee7e9698c 2023-04-24 18:56:09,417 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/16ec637afd424e1893237992b9d968ab to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/16ec637afd424e1893237992b9d968ab 2023-04-24 18:56:09,418 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c1e03e7e44b44a5295d721bfaf58a551 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/C/c1e03e7e44b44a5295d721bfaf58a551 2023-04-24 18:56:09,420 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/recovered.edits/735.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603/recovered.edits/735.seqid 2023-04-24 18:56:09,420 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/55e964c86d584c4edcab7a055c74c603 2023-04-24 18:56:09,420 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:56:09,422 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,423 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:56:09,424 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:56:09,425 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,425 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:56:09,425 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362569425"}]},"ts":"9223372036854775807"} 2023-04-24 18:56:09,426 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:56:09,427 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 55e964c86d584c4edcab7a055c74c603, NAME => 'TestAcidGuarantees,,1682362540622.55e964c86d584c4edcab7a055c74c603.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:56:09,427 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:56:09,427 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362569427"}]},"ts":"9223372036854775807"} 2023-04-24 18:56:09,428 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:56:09,429 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,430 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-24 18:56:09,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-24 18:56:09,503 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-24 18:56:09,515 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 229), OpenFileDescriptor=563 (was 535) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=376 (was 388), ProcessCount=170 (was 170), AvailableMemoryMB=4579 (was 4575) - AvailableMemoryMB LEAK? - 2023-04-24 18:56:09,529 INFO [Listener at localhost/43631] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=376, ProcessCount=170, AvailableMemoryMB=4578 2023-04-24 18:56:09,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:56:09,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:56:09,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:09,533 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 18:56:09,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-24 18:56:09,534 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 18:56:09,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 18:56:09,535 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:09,535 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e empty. 2023-04-24 18:56:09,536 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:09,536 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:56:09,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 18:56:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 18:56:09,942 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:56:09,944 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1a5268f22ea5dbabf8eac37708fa245e, NAME => 'TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp 2023-04-24 18:56:09,948 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:56:09,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1a5268f22ea5dbabf8eac37708fa245e, disabling compactions & flushes 2023-04-24 18:56:09,949 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:09,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:09,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. after waiting 0 ms 2023-04-24 18:56:09,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:09,949 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:09,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:09,950 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 18:56:09,951 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362569951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362569951"}]},"ts":"1682362569951"} 2023-04-24 18:56:09,952 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 18:56:09,953 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 18:56:09,953 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362569953"}]},"ts":"1682362569953"} 2023-04-24 18:56:09,954 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 18:56:09,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, ASSIGN}] 2023-04-24 18:56:09,958 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, ASSIGN 2023-04-24 18:56:09,958 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=false 2023-04-24 18:56:10,109 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:10,110 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362570109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362570109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362570109"}]},"ts":"1682362570109"} 2023-04-24 18:56:10,111 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:56:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 18:56:10,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:10,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a5268f22ea5dbabf8eac37708fa245e, NAME => 'TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:56:10,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:56:10,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,267 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,268 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A 2023-04-24 18:56:10,268 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A 2023-04-24 18:56:10,268 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:10,268 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName A 2023-04-24 18:56:10,269 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:10,269 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,270 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B 2023-04-24 18:56:10,270 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B 2023-04-24 18:56:10,270 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:10,270 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName B 2023-04-24 18:56:10,270 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:10,270 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,272 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C 2023-04-24 18:56:10,272 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C 2023-04-24 18:56:10,272 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:10,272 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName C 2023-04-24 18:56:10,272 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:10,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:10,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:56:10,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:10,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 18:56:10,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1a5268f22ea5dbabf8eac37708fa245e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68653949, jitterRate=0.023023560643196106}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:56:10,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:10,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., pid=56, masterSystemTime=1682362570262 2023-04-24 18:56:10,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:10,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:10,279 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:10,280 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362570279"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362570279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362570279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362570279"}]},"ts":"1682362570279"} 2023-04-24 18:56:10,281 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-24 18:56:10,281 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 in 169 msec 2023-04-24 18:56:10,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-24 18:56:10,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, ASSIGN in 325 msec 2023-04-24 18:56:10,283 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 18:56:10,283 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362570283"}]},"ts":"1682362570283"} 2023-04-24 18:56:10,284 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 18:56:10,286 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 18:56:10,287 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 755 msec 2023-04-24 18:56:10,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 18:56:10,638 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-24 18:56:10,639 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:10,643 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:10,644 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:10,646 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:10,646 DEBUG [Listener at localhost/43631] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 18:56:10,648 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 18:56:10,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 18:56:10,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 18:56:10,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:11,059 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:56:11,061 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 18:56:11,061 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 18:56:11,062 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 18:56:11,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, REOPEN/MOVE}] 2023-04-24 18:56:11,065 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, REOPEN/MOVE 2023-04-24 18:56:11,066 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,066 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362571066"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362571066"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362571066"}]},"ts":"1682362571066"} 2023-04-24 18:56:11,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:56:11,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1a5268f22ea5dbabf8eac37708fa245e, disabling compactions & flushes 2023-04-24 18:56:11,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. after waiting 0 ms 2023-04-24 18:56:11,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 18:56:11,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:11,223 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1a5268f22ea5dbabf8eac37708fa245e to self. 2023-04-24 18:56:11,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,224 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=CLOSED 2023-04-24 18:56:11,224 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362571224"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362571224"}]},"ts":"1682362571224"} 2023-04-24 18:56:11,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-24 18:56:11,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 in 159 msec 2023-04-24 18:56:11,227 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37593,1682362415680; forceNewPlan=false, retain=true 2023-04-24 18:56:11,377 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,377 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362571377"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362571377"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362571377"}]},"ts":"1682362571377"} 2023-04-24 18:56:11,378 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:56:11,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a5268f22ea5dbabf8eac37708fa245e, NAME => 'TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.', STARTKEY => '', ENDKEY => ''} 2023-04-24 18:56:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 18:56:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,534 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,535 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A 2023-04-24 18:56:11,535 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A 2023-04-24 18:56:11,535 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:11,536 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName A 2023-04-24 18:56:11,537 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:11,537 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,537 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B 2023-04-24 18:56:11,538 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B 2023-04-24 18:56:11,538 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:11,538 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName B 2023-04-24 18:56:11,538 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:11,538 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,539 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C 2023-04-24 18:56:11,539 DEBUG [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C 2023-04-24 18:56:11,539 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 18:56:11,539 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a5268f22ea5dbabf8eac37708fa245e columnFamilyName C 2023-04-24 18:56:11,539 INFO [StoreOpener-1a5268f22ea5dbabf8eac37708fa245e-1] regionserver.HStore(310): Store=1a5268f22ea5dbabf8eac37708fa245e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 18:56:11,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 18:56:11,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1a5268f22ea5dbabf8eac37708fa245e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64248322, jitterRate=-0.04262539744377136}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 18:56:11,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:11,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., pid=61, masterSystemTime=1682362571530 2023-04-24 18:56:11,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,546 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,546 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362571546"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682362571546"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682362571546"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682362571546"}]},"ts":"1682362571546"} 2023-04-24 18:56:11,548 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-24 18:56:11,548 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 in 169 msec 2023-04-24 18:56:11,549 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-24 18:56:11,549 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, REOPEN/MOVE in 483 msec 2023-04-24 18:56:11,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-24 18:56:11,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-24 18:56:11,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-24 18:56:11,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-24 18:56:11,554 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,558 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,559 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,562 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,563 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,566 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,566 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,571 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,571 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,575 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,575 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,579 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,579 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,583 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,583 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,593 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,594 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,597 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,598 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:59583 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 18:56:11,606 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 18:56:11,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:11,614 DEBUG [hconnection-0x705b0e42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,614 DEBUG [hconnection-0x4fed9901-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,614 DEBUG [hconnection-0x84abc28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,615 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,616 DEBUG [hconnection-0x3f9a48a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,621 DEBUG [hconnection-0x736f5d5b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,621 DEBUG [hconnection-0x5278ac1e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,621 DEBUG [hconnection-0x74e96c9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,622 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,625 DEBUG [hconnection-0x259280ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,626 DEBUG [hconnection-0x599b44b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,626 DEBUG [hconnection-0x7ca90553-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 18:56:11,627 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,628 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,629 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,629 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,629 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,630 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,638 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 18:56:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:11,641 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:11,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:11,641 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:11,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:11,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:56:11,643 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:11,643 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:11,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:11,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:11,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:11,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:11,648 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,648 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,649 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:11,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:11,658 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:11,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f58e7c5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:11,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:11,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,661 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:11,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,661 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:11,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:11,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:11,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:11,665 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:11,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:11,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:11,665 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:11,665 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:11,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,667 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,668 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,668 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424323ed58384234d6abdf736446bc0f145 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424323ed58384234d6abdf736446bc0f145 2023-04-24 18:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,671 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:11,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,672 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:11,672 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:11,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362571676, max timestamp=1682362571681Num uniques 859; 2023-04-24 18:56:11,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:11,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362571676, max timestamp=1682362571681Num uniques 859; 2023-04-24 18:56:11,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:11,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362571676, max timestamp=1682362571681Num uniques 859; 2023-04-24 18:56:11,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:11,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362631692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362631693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362631694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362631695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362631695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,772 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:11,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:11,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:11,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:11,776 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:11,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:11,776 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:11,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,778 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,778 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:11,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:11,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:11,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:11,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:11,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:11,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:11,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:11,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:11,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:11,784 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,785 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,785 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,785 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:11,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382002c4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:11,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:11,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:11,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,789 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:11,791 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:11,794 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:11,794 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:11,794 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:11,794 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:11,794 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:11,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,797 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362631796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362631797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,800 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:11,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:11,800 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,801 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,801 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:11,801 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:11,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362631798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362631799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:11,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362631799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,901 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:11,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:11,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:11,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:11,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:11,905 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:11,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:11,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:11,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,907 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,907 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,908 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:11,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:11,909 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,909 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:11,909 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:11,909 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:11,910 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,910 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:11,910 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:11,911 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457447bd[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:11,913 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:11,913 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,914 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:11,915 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,915 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:11,916 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,916 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,916 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,916 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:11,918 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:11,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:11,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:11,918 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:11,918 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:11,921 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,921 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,921 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:11,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,921 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:11,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,922 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:11,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:11,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:11,925 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:11,925 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:11,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:11,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:11,925 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:11,925 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:11,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:11,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362632000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362632000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362632002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362632004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362632003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,025 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,029 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,029 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,031 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,033 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,033 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,033 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,033 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:12,033 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,038 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d7c697[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:12,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:12,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,042 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:12,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:12,042 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:12,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,042 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:12,044 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:12,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:12,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:12,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:12,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:12,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:12,049 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,049 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,049 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,049 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,051 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,053 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,053 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:12,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:12,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:12,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,054 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:12,054 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:12,074 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/fac8e019a35a48a981329ea5a97ce9af 2023-04-24 18:56:12,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/4d935bbf93c740979760296f00417c0f 2023-04-24 18:56:12,154 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,157 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,158 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,159 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,159 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,161 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:12,163 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae979c7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:12,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:12,165 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:12,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:12,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:12,166 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:12,168 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:12,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:12,171 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:12,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:12,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:12,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:12,171 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:12,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,173 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,176 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:12,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:12,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,177 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,177 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:12,177 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:12,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:12,277 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,281 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,281 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,284 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,286 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:12,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:12,287 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,287 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e7707c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:12,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:12,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,291 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:12,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:12,292 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:12,295 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:12,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:12,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:12,295 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:12,295 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:12,298 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,298 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,298 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,299 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,302 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:12,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,303 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,303 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:12,303 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:12,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362632302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:12,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362632302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362632306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362632307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362632308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,403 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,406 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,407 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,408 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,410 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,410 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,410 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,410 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:12,410 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:12,412 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c98c32[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:12,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:12,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,419 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:12,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,419 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:12,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:12,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:12,420 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:12,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:12,423 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:12,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:12,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:12,423 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:12,423 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:12,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,425 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,427 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:12,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,431 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:12,431 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,431 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,432 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,532 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,535 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,536 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,536 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/556f290a8b284e8f88401433dbdaee13 2023-04-24 18:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,538 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,539 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,539 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:12,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:12,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:12,540 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:12,541 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,541 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/fac8e019a35a48a981329ea5a97ce9af as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af 2023-04-24 18:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39260762[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:12,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:12,543 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,544 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,544 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:12,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af, entries=150, sequenceid=16, filesize=25.1 K 2023-04-24 18:56:12,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/4d935bbf93c740979760296f00417c0f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f 2023-04-24 18:56:12,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f, entries=150, sequenceid=16, filesize=11.6 K 2023-04-24 18:56:12,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/556f290a8b284e8f88401433dbdaee13 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13 2023-04-24 18:56:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:12,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13, entries=150, sequenceid=16, filesize=11.6 K 2023-04-24 18:56:12,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 1a5268f22ea5dbabf8eac37708fa245e in 912ms, sequenceid=16, compaction requested=false 2023-04-24 18:56:12,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:12,558 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:12,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:12,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:12,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:12,559 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:12,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:12,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,563 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:12,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:12,565 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:12,565 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:12,565 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:12,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:12,665 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:12,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:12,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:12,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:12,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:12,669 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:12,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:12,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:12,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,670 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:12,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:12,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:12,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:12,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:12,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:12,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:12,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:12,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:12,672 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:12,672 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:12,672 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:12,672 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:12,673 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:12,690 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424bae273b17d4b4b36af790358ef2aea6a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bae273b17d4b4b36af790358ef2aea6a 2023-04-24 18:56:12,701 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/8d7ecf1404204d83aa85e7d78b3b479b 2023-04-24 18:56:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:12,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:12,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362632814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362632815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362632816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362632817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362632817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362632919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362632919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362632919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362632921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:12,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362632921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,115 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/2a1bbc566648429d9b2dc94c6554a42d 2023-04-24 18:56:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362633124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362633124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362633124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362633124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362633125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362633427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362633428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362633428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362633429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362633429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,526 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/bd92e09724d943d3a4c7910844772984 2023-04-24 18:56:13,530 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/8d7ecf1404204d83aa85e7d78b3b479b as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b 2023-04-24 18:56:13,533 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b, entries=150, sequenceid=51, filesize=25.1 K 2023-04-24 18:56:13,534 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/2a1bbc566648429d9b2dc94c6554a42d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d 2023-04-24 18:56:13,537 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d, entries=150, sequenceid=51, filesize=11.6 K 2023-04-24 18:56:13,538 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/bd92e09724d943d3a4c7910844772984 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984 2023-04-24 18:56:13,545 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984, entries=150, sequenceid=51, filesize=11.6 K 2023-04-24 18:56:13,545 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 1a5268f22ea5dbabf8eac37708fa245e in 873ms, sequenceid=51, compaction requested=false 2023-04-24 18:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:13,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:13,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,549 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:13,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a076c0[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:13,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:13,551 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:13,554 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:13,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:13,556 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:13,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:13,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:13,556 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:13,557 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:13,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:13,558 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:13,559 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:13,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:13,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:13,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:13,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:13,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:13,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:13,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,568 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:13,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:13,568 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:13,668 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:13,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:13,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:13,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:13,671 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:13,671 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:13,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:13,673 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:13,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:13,673 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,673 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:13,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:13,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:13,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:13,675 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:13,675 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:13,675 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:56:13,675 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:13,676 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:13,676 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:13,676 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:13,676 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:13,676 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:13,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:13,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:13,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362633947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362633947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362633948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362573942, max timestamp=1682362573948Num uniques 859; 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362573942, max timestamp=1682362573948Num uniques 859; 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682362573942, max timestamp=1682362573948Num uniques 859; 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:13,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:13,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362633978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:13,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:13,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362633982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362634051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362634051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362634051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362634083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362634087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,091 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304248e4c49d245094201ad1ef695d20baccb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248e4c49d245094201ad1ef695d20baccb 2023-04-24 18:56:14,096 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/97fa5e14ca00475192cd8c6f6fe781a6 2023-04-24 18:56:14,105 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/054fad0f1e4f43f7a51f4b0db59b0ece 2023-04-24 18:56:14,114 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/38f097910d4b4447b70e800431122aa4 2023-04-24 18:56:14,119 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/97fa5e14ca00475192cd8c6f6fe781a6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6 2023-04-24 18:56:14,122 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6, entries=150, sequenceid=63, filesize=25.1 K 2023-04-24 18:56:14,123 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/054fad0f1e4f43f7a51f4b0db59b0ece as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece 2023-04-24 18:56:14,126 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece, entries=150, sequenceid=63, filesize=11.6 K 2023-04-24 18:56:14,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/38f097910d4b4447b70e800431122aa4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4 2023-04-24 18:56:14,130 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4, entries=150, sequenceid=63, filesize=11.6 K 2023-04-24 18:56:14,131 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 1a5268f22ea5dbabf8eac37708fa245e in 456ms, sequenceid=63, compaction requested=true 2023-04-24 18:56:14,131 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:14,131 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,131 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712d8138[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,135 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,136 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,137 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,137 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,139 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,142 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,142 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,143 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,146 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,150 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,150 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,151 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,151 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,251 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,255 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,255 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,255 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,255 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,257 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 18:56:14,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,257 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:14,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,259 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:14,259 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:14,259 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:14,259 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,262 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87c2b33[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,265 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,265 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,266 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,266 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,268 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362634266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,270 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,270 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,271 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,271 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362634268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362634269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,272 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,272 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,272 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,273 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,274 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,277 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,277 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,277 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,278 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,278 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,278 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362634287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362634290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,362 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 18:56:14,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362634370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362634373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362634373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,378 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,383 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,383 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,384 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,384 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:14,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,394 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,394 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,394 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,394 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,395 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,396 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e82049[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,398 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,400 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,400 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,406 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,406 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,406 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,409 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,413 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,413 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,414 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,414 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,514 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,517 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,518 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,519 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,519 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,520 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,520 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:14,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,522 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,522 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,523 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,523 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,525 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,525 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae550c2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,527 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,528 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,529 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,530 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,533 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,533 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,533 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,533 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,533 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,535 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,535 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,537 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,545 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,545 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,546 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,546 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,546 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362634574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362634576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362634576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362634591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362634594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,646 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,650 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,650 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,650 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,650 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,651 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,652 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,652 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,653 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:14,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,654 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,654 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,654 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,654 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,656 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c26e7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,660 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,662 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,662 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,662 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,663 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,663 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,666 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,666 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,668 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,669 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,670 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,672 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,672 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,673 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,673 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,673 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,681 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d24b9c414a1d4efe9a9c617027c23e4c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d24b9c414a1d4efe9a9c617027c23e4c 2023-04-24 18:56:14,773 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,777 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,777 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,779 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,779 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,783 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,783 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e90b08c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,786 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,787 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,787 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,787 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,789 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,792 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,792 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,792 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,794 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,795 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,795 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,796 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,799 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,799 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,800 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,800 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:14,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362634876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362634880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362634880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,900 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:14,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:14,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:14,903 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:14,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:14,903 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:14,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,904 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,905 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,905 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:14,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:14,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:14,907 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:14,909 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,909 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:14,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba2f7f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:14,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:14,911 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,913 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:14,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:14,913 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:14,914 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,914 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:14,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:14,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,915 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,915 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:14,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:14,917 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:14,917 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:14,917 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:14,917 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:14,917 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:14,919 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,919 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:14,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,919 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:14,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,919 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,921 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:14,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:14,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:14,926 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:14,927 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:14,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:14,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:14,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:14,927 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:14,927 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:15,027 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:15,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:15,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:15,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:15,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:15,031 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:15,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:15,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:15,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,032 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,033 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,033 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:15,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:15,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:15,035 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,035 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:15,035 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:15,035 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:15,036 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,036 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,037 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e69092a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:15,039 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:15,039 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,041 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:15,041 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:15,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,041 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:15,041 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:15,043 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:15,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,043 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:15,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:15,045 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:15,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:15,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:15,046 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:15,046 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:15,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,047 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,048 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,049 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,051 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,051 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,052 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:15,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,052 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,052 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:15,052 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:15,086 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/60538da35a0b401b88159de3cf995e43 2023-04-24 18:56:15,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/ead48e4519414fc6b5492c6c6abcac71 2023-04-24 18:56:15,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362635098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362635100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,152 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:15,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:15,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:15,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:15,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:15,156 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,156 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,162 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,162 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,162 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,162 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:15,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:15,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,163 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:15,164 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,164 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:15,164 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:15,164 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:15,164 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:15,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,166 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78e46f1e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:15,168 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:15,168 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,169 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,169 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:15,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:15,169 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:15,170 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:15,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:15,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,171 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:15,174 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:15,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:15,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:15,174 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:15,174 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:15,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,176 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,177 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,178 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:15,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:15,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,181 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:15,181 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:15,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:15,181 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,182 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,182 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,182 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,182 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,282 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:15,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:15,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:15,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:15,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:15,285 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:15,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:15,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:15,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,286 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,286 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,287 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:15,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:15,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:15,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:15,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:15,288 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,288 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:15,290 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,290 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,290 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139ba45e[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:15,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:15,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,293 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,293 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:15,293 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:15,293 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:15,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,294 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:15,295 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,295 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:15,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:15,300 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:15,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:15,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:15,300 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:15,300 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:15,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,302 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,303 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,304 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:15,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,306 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:15,306 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,307 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:15,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,307 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:15,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362635382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:15,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362635383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362635384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,407 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:15,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:15,411 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:15,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:15,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:15,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,411 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,416 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,416 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,416 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,417 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:15,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:15,419 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,419 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:15,419 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:15,419 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:15,423 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca5b085[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:15,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:15,425 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,426 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:15,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:15,428 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:15,430 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:15,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:15,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:15,431 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:15,431 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:15,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:15,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,433 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:15,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,434 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,436 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,438 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:15,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,439 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:15,439 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:15,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:15,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:15,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/8483b0ed40614c6aa6e7c37b252925ef 2023-04-24 18:56:15,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/60538da35a0b401b88159de3cf995e43 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43 2023-04-24 18:56:15,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43, entries=150, sequenceid=99, filesize=25.1 K 2023-04-24 18:56:15,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/ead48e4519414fc6b5492c6c6abcac71 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71 2023-04-24 18:56:15,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71, entries=150, sequenceid=99, filesize=11.6 K 2023-04-24 18:56:15,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/8483b0ed40614c6aa6e7c37b252925ef as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef 2023-04-24 18:56:15,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef, entries=150, sequenceid=99, filesize=11.6 K 2023-04-24 18:56:15,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1a5268f22ea5dbabf8eac37708fa245e in 1264ms, sequenceid=99, compaction requested=true 2023-04-24 18:56:15,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:15,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:15,522 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:56:15,522 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:56:15,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:15,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:15,523 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:56:15,523 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:15,523 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,523 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=46.4 K 2023-04-24 18:56:15,524 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:56:15,524 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:15,524 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,524 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=100.6 K 2023-04-24 18:56:15,524 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4d935bbf93c740979760296f00417c0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682362571631 2023-04-24 18:56:15,524 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fac8e019a35a48a981329ea5a97ce9af, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682362571631 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2a1bbc566648429d9b2dc94c6554a42d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682362571686 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8d7ecf1404204d83aa85e7d78b3b479b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682362571686 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 054fad0f1e4f43f7a51f4b0db59b0ece, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682362572807 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 97fa5e14ca00475192cd8c6f6fe781a6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682362572807 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting ead48e4519414fc6b5492c6c6abcac71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:15,525 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 60538da35a0b401b88159de3cf995e43, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:15,532 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:15,538 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#511 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:15,539 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:15,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:15,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:15,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:15,546 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:15,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:15,546 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:15,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,547 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,548 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:15,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:15,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:15,548 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:15,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:15,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:15,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:15,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:15,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:15,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:15,580 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:15,580 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:15,580 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,580 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:15,580 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:15,581 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:15,608 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304245b147fd0a4c84071b5aae601951c2a55 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245b147fd0a4c84071b5aae601951c2a55 2023-04-24 18:56:15,614 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/06e0b4f825074d52a1971f6d9753fe85 2023-04-24 18:56:15,623 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/b97ca1d4e2b445e0acc1020740bdb99a 2023-04-24 18:56:15,951 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/26fc198e156745e196c6e1c77b6be391 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/26fc198e156745e196c6e1c77b6be391 2023-04-24 18:56:15,955 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into 26fc198e156745e196c6e1c77b6be391(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:15,955 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:15,955 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=12, startTime=1682362575522; duration=0sec 2023-04-24 18:56:15,955 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:15,955 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 18:56:15,956 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 18:56:15,956 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:15,956 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:15,957 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=46.4 K 2023-04-24 18:56:15,957 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 556f290a8b284e8f88401433dbdaee13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682362571631 2023-04-24 18:56:15,957 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting bd92e09724d943d3a4c7910844772984, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682362571686 2023-04-24 18:56:15,957 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 38f097910d4b4447b70e800431122aa4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682362572807 2023-04-24 18:56:15,958 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8483b0ed40614c6aa6e7c37b252925ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:15,964 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:15,973 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/e63348c6358b4288b546978a7a04ed34 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/e63348c6358b4288b546978a7a04ed34 2023-04-24 18:56:15,978 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into e63348c6358b4288b546978a7a04ed34(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:15,978 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:15,978 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=12, startTime=1682362575522; duration=0sec 2023-04-24 18:56:15,978 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:16,084 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/3d50e3d8488e4764addb639560d5afe4 2023-04-24 18:56:16,087 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/06e0b4f825074d52a1971f6d9753fe85 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85 2023-04-24 18:56:16,091 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85, entries=100, sequenceid=109, filesize=18.4 K 2023-04-24 18:56:16,091 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/b97ca1d4e2b445e0acc1020740bdb99a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a 2023-04-24 18:56:16,095 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a, entries=100, sequenceid=109, filesize=9.3 K 2023-04-24 18:56:16,095 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/3d50e3d8488e4764addb639560d5afe4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4 2023-04-24 18:56:16,099 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4, entries=100, sequenceid=109, filesize=9.3 K 2023-04-24 18:56:16,099 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 1a5268f22ea5dbabf8eac37708fa245e in 519ms, sequenceid=109, compaction requested=false 2023-04-24 18:56:16,100 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:16,100 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,101 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a53d462[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,111 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,116 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,116 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,116 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,117 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,117 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,117 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,122 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,122 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:16,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:16,138 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362576152, max timestamp=1682362576174Num uniques 859; 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362576152, max timestamp=1682362576174Num uniques 859; 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362576152, max timestamp=1682362576174Num uniques 859; 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:16,181 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424790a83bc1c5340d2a571ce6d6f8b7fcb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424790a83bc1c5340d2a571ce6d6f8b7fcb 2023-04-24 18:56:16,187 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/1f9e01f1551e446d9055a5b99b696564 2023-04-24 18:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362576196, max timestamp=1682362576212Num uniques 1018; 2023-04-24 18:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682362576196, max timestamp=1682362576212Num uniques 1018; 2023-04-24 18:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 18:56:16,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682362576196, max timestamp=1682362576212Num uniques 1018; 2023-04-24 18:56:16,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-24 18:56:16,222 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,226 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,229 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,231 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,231 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,231 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,231 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,233 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6194d0a9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,234 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,234 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,237 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,239 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,239 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,247 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,247 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362636262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362636263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,347 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,351 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,353 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,360 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe33191[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,363 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362636366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362636367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,368 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,368 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,370 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,370 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,373 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,373 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,374 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,375 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,376 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,380 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,381 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,381 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,389 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/883e8cb986624d999a9a913d9b7f4fc5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/883e8cb986624d999a9a913d9b7f4fc5 2023-04-24 18:56:16,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362636388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,393 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 883e8cb986624d999a9a913d9b7f4fc5(size=25.3 K), total size for store is 43.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:16,393 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:16,393 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=12, startTime=1682362575522; duration=0sec 2023-04-24 18:56:16,394 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:16,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362636394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362636394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,481 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,485 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,486 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,491 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e039d70[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,493 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,493 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,495 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,498 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,498 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,501 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,504 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,505 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,505 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362636569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362636569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/8c2144dd041e4da8861dde35de94fd51 2023-04-24 18:56:16,605 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,609 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,609 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,610 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,610 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,611 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,612 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,612 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,613 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,615 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a57a416[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,617 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,619 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,620 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,623 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,623 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,630 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,630 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,731 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,734 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,734 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,736 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,736 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,736 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,736 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,738 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,738 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,739 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d81346d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,742 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,745 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,748 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,748 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,748 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,754 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,756 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,756 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,756 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,756 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,857 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,860 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,860 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,861 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,862 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,862 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,863 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,865 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,865 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,865 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,865 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,867 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b58e3f5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,871 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:16,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,871 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:16,872 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362636873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:16,875 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:16,875 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:16,875 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:16,875 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:16,875 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:16,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362636873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,877 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,879 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:16,881 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,881 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:16,881 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:16,881 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:16,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:16,982 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:16,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:16,985 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:16,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:16,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:16,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,988 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:16,993 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:16,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:16,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:16,995 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:16,995 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,997 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:16,997 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c201f3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:16,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:16,999 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:17,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:17,001 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:17,002 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:17,001 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,002 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:17,004 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:17,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:17,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:17,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:17,004 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:17,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/3e9e77ef48bb40c28919677e00b86c7e 2023-04-24 18:56:17,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:17,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:17,013 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/1f9e01f1551e446d9055a5b99b696564 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564 2023-04-24 18:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:17,014 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564, entries=350, sequenceid=121, filesize=52.3 K 2023-04-24 18:56:17,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:17,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:17,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:17,017 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:17,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:17,017 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:17,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/8c2144dd041e4da8861dde35de94fd51 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,018 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:17,018 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:17,018 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:17,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51, entries=150, sequenceid=121, filesize=11.6 K 2023-04-24 18:56:17,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/3e9e77ef48bb40c28919677e00b86c7e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e 2023-04-24 18:56:17,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e, entries=150, sequenceid=121, filesize=11.6 K 2023-04-24 18:56:17,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.14 KB/134292 for 1a5268f22ea5dbabf8eac37708fa245e in 889ms, sequenceid=121, compaction requested=true 2023-04-24 18:56:17,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:17,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:17,027 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:17,027 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:17,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:17,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:17,028 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98192 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:17,028 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:17,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:17,028 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:17,028 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:17,028 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/883e8cb986624d999a9a913d9b7f4fc5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=95.9 K 2023-04-24 18:56:17,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:17,028 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/26fc198e156745e196c6e1c77b6be391, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=32.7 K 2023-04-24 18:56:17,028 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 883e8cb986624d999a9a913d9b7f4fc5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:17,029 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 06e0b4f825074d52a1971f6d9753fe85, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362574267 2023-04-24 18:56:17,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 26fc198e156745e196c6e1c77b6be391, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:17,029 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1f9e01f1551e446d9055a5b99b696564, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576114 2023-04-24 18:56:17,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b97ca1d4e2b445e0acc1020740bdb99a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362574267 2023-04-24 18:56:17,029 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 8c2144dd041e4da8861dde35de94fd51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576131 2023-04-24 18:56:17,034 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#520 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:17,045 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#519 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:17,052 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/4d64e2b514ca40f0a1ad0961a3ad108e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d64e2b514ca40f0a1ad0961a3ad108e 2023-04-24 18:56:17,056 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into 4d64e2b514ca40f0a1ad0961a3ad108e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:17,056 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:17,056 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=13, startTime=1682362577027; duration=0sec 2023-04-24 18:56:17,057 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:17,057 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:17,057 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:17,058 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:17,058 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:17,058 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/e63348c6358b4288b546978a7a04ed34, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=32.7 K 2023-04-24 18:56:17,058 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting e63348c6358b4288b546978a7a04ed34, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682362573960 2023-04-24 18:56:17,058 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3d50e3d8488e4764addb639560d5afe4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682362574267 2023-04-24 18:56:17,059 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3e9e77ef48bb40c28919677e00b86c7e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576131 2023-04-24 18:56:17,063 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#521 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:17,118 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:17,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:17,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:17,121 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:17,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:17,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:17,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:17,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:17,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:17,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,124 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:17,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:17,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:17,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:17,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:17,126 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:17,126 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:17,126 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=131.14 KB heapSize=372.55 KB 2023-04-24 18:56:17,126 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:17,126 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:17,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:17,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:17,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:17,127 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:17,146 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304240b86c9471afb41de953975cb9c90dbcc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240b86c9471afb41de953975cb9c90dbcc 2023-04-24 18:56:17,162 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/a1e05512edcd4e6f9b802671df047a9f 2023-04-24 18:56:17,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:17,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:17,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362637417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362637418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,481 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/3d66d1a769544e4f93abc7b990659dc4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d66d1a769544e4f93abc7b990659dc4 2023-04-24 18:56:17,484 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into 3d66d1a769544e4f93abc7b990659dc4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:17,485 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:17,485 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=13, startTime=1682362577027; duration=0sec 2023-04-24 18:56:17,485 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:17,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362637522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362637522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,581 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/56f77a1e96a343f9a57edc237d49a976 2023-04-24 18:56:17,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362637725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362637726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:17,869 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f 2023-04-24 18:56:17,872 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 5aa3c4d8a2bf4b38b65ffb8b38f4a98f(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:17,872 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:17,872 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=13, startTime=1682362577027; duration=0sec 2023-04-24 18:56:17,872 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:17,990 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=41.69 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/65f515c6f037423fba2be8a5965fff13 2023-04-24 18:56:17,994 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/a1e05512edcd4e6f9b802671df047a9f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f 2023-04-24 18:56:17,997 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f, entries=150, sequenceid=176, filesize=25.3 K 2023-04-24 18:56:17,998 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/56f77a1e96a343f9a57edc237d49a976 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976 2023-04-24 18:56:18,001 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976, entries=150, sequenceid=176, filesize=11.8 K 2023-04-24 18:56:18,002 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/65f515c6f037423fba2be8a5965fff13 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13 2023-04-24 18:56:18,005 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13, entries=150, sequenceid=176, filesize=11.8 K 2023-04-24 18:56:18,006 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.14 KB/134292, heapSize ~372.50 KB/381440, currentSize=46.96 KB/48090 for 1a5268f22ea5dbabf8eac37708fa245e in 880ms, sequenceid=176, compaction requested=false 2023-04-24 18:56:18,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:18,006 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,006 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec27bc9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,010 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,012 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,012 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,014 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,015 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,015 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,015 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,019 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,019 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,019 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,021 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,028 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,031 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:18,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:18,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,040 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,040 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,040 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,040 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,041 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362578032, max timestamp=1682362578061Num uniques 859; 2023-04-24 18:56:18,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:18,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362578032, max timestamp=1682362578061Num uniques 859; 2023-04-24 18:56:18,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:18,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362578053, max timestamp=1682362578061Num uniques 859; 2023-04-24 18:56:18,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:18,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362638090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362638091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,140 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,145 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,146 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,147 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,147 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,148 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,149 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,150 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,151 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,152 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ae41220[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,154 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,156 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,157 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,161 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,164 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,164 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,164 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,166 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,167 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,169 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,171 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,171 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,172 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,172 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,172 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362638194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362638195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,272 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,275 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,275 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,276 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,277 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,277 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,279 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,279 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,282 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,282 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,283 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46df09c2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,284 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,285 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,285 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,285 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,286 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,289 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,289 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,289 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,290 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,291 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,293 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,296 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,297 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,296 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,297 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,297 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,397 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362638398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362638398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,401 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,401 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,402 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,403 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,403 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362638402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,405 DEBUG [Thread-3762] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,407 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,407 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,407 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,407 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,407 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,408 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362638407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,411 DEBUG [Thread-3764] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:18,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362638408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,412 DEBUG [Thread-3766] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fcb4573[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,414 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,416 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,416 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,416 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,423 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,426 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,426 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,426 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,429 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,429 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,429 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,445 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,445 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,445 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,445 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,473 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424beeafdcca6a3483c9432c18916f7d01c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424beeafdcca6a3483c9432c18916f7d01c 2023-04-24 18:56:18,478 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/ad4f643770284967b10801404cfd328d 2023-04-24 18:56:18,546 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,549 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,549 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,550 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,552 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,552 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,553 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,554 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,554 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,554 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,554 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,554 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,556 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327a6970[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,558 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,560 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,560 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,561 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,564 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,564 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,564 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,566 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,577 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,577 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,578 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,678 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,681 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,681 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,683 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,683 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,684 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,685 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,685 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,685 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,685 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,687 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2895180d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,696 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,697 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,700 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,700 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,700 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362638701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,703 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,705 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362638703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,707 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,708 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,708 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,707 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,708 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,808 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,812 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,812 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,818 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,818 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:18,820 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:18,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@163a95c4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:18,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:18,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:18,825 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,825 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:18,826 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:18,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:18,832 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:18,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:18,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:18,832 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:18,832 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:18,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,839 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:18,840 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,843 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:18,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,844 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:18,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:18,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:18,844 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,844 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:18,844 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:18,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/519048f9949540e68033580a8de263d5 2023-04-24 18:56:18,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/5e2b0b79199b4670af7349eb253e9bf0 2023-04-24 18:56:18,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/ad4f643770284967b10801404cfd328d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d 2023-04-24 18:56:18,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d, entries=150, sequenceid=190, filesize=25.3 K 2023-04-24 18:56:18,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/519048f9949540e68033580a8de263d5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5 2023-04-24 18:56:18,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5, entries=150, sequenceid=190, filesize=11.8 K 2023-04-24 18:56:18,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/5e2b0b79199b4670af7349eb253e9bf0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0 2023-04-24 18:56:18,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0, entries=150, sequenceid=190, filesize=11.8 K 2023-04-24 18:56:18,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.53 KB/121370 for 1a5268f22ea5dbabf8eac37708fa245e in 893ms, sequenceid=190, compaction requested=true 2023-04-24 18:56:18,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:18,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:18,924 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:18,924 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:18,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:18,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:18,925 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77778 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:18,925 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:18,926 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,926 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=76.0 K 2023-04-24 18:56:18,926 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:18,926 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:18,926 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,926 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d64e2b514ca40f0a1ad0961a3ad108e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.3 K 2023-04-24 18:56:18,926 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5aa3c4d8a2bf4b38b65ffb8b38f4a98f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576131 2023-04-24 18:56:18,926 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4d64e2b514ca40f0a1ad0961a3ad108e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576131 2023-04-24 18:56:18,926 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a1e05512edcd4e6f9b802671df047a9f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682362576255 2023-04-24 18:56:18,927 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting ad4f643770284967b10801404cfd328d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:18,927 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 56f77a1e96a343f9a57edc237d49a976, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682362576255 2023-04-24 18:56:18,927 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 519048f9949540e68033580a8de263d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:18,933 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:18,937 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#528 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:18,944 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:18,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:18,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:18,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:18,952 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:18,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:18,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:18,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,952 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,953 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,953 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:18,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:18,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:18,954 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:18,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:18,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:18,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:18,964 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=118.53 KB heapSize=335.38 KB 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:18,964 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:18,983 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424018d442d4fa846ffa42d6aeb8543a7e5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424018d442d4fa846ffa42d6aeb8543a7e5 2023-04-24 18:56:19,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:19,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362639218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34354 deadline: 1682362639219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362639322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362639324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,349 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/02e7a3ec13b941cb9d64e8dda84b6788 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/02e7a3ec13b941cb9d64e8dda84b6788 2023-04-24 18:56:19,353 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into 02e7a3ec13b941cb9d64e8dda84b6788(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:19,353 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:19,353 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=13, startTime=1682362578924; duration=0sec 2023-04-24 18:56:19,353 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:19,353 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:19,354 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:19,354 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:19,354 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:19,355 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d66d1a769544e4f93abc7b990659dc4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.3 K 2023-04-24 18:56:19,355 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3d66d1a769544e4f93abc7b990659dc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682362576131 2023-04-24 18:56:19,355 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 65f515c6f037423fba2be8a5965fff13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682362576255 2023-04-24 18:56:19,356 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5e2b0b79199b4670af7349eb253e9bf0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:19,362 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:19,388 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/8aec85c73fb44d8f8fc0bc68b93f62c7 2023-04-24 18:56:19,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362639525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362639528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,778 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/220d02ae5b9f437d8a32cd3c5974217f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/220d02ae5b9f437d8a32cd3c5974217f 2023-04-24 18:56:19,784 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into 220d02ae5b9f437d8a32cd3c5974217f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:19,784 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:19,784 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=13, startTime=1682362578925; duration=0sec 2023-04-24 18:56:19,784 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:19,789 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/8e240e57c0d94418a12dadf5d9cb8cb7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8e240e57c0d94418a12dadf5d9cb8cb7 2023-04-24 18:56:19,793 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 8e240e57c0d94418a12dadf5d9cb8cb7(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:19,793 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:19,793 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=13, startTime=1682362578924; duration=0sec 2023-04-24 18:56:19,793 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:19,815 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/4227d7a295394c808d88216d67d51eb6 2023-04-24 18:56:19,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34412 deadline: 1682362639829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:19,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362639831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,225 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/7c8cc1900cec46b9a30468ee6027c369 2023-04-24 18:56:20,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/8aec85c73fb44d8f8fc0bc68b93f62c7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7 2023-04-24 18:56:20,231 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7, entries=150, sequenceid=224, filesize=25.3 K 2023-04-24 18:56:20,232 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/4227d7a295394c808d88216d67d51eb6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6 2023-04-24 18:56:20,236 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6, entries=150, sequenceid=224, filesize=11.8 K 2023-04-24 18:56:20,237 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/7c8cc1900cec46b9a30468ee6027c369 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369 2023-04-24 18:56:20,239 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369, entries=150, sequenceid=224, filesize=11.8 K 2023-04-24 18:56:20,240 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~335.33 KB/343376, currentSize=60.38 KB/61830 for 1a5268f22ea5dbabf8eac37708fa245e in 1276ms, sequenceid=224, compaction requested=false 2023-04-24 18:56:20,240 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:20,240 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539faaad[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,248 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,250 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,253 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,253 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,253 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,255 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,255 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,255 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,256 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,258 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,261 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,261 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,261 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,261 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,262 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:20,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:20,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,351 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ede54afae93944ee8178845436d3644e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ede54afae93944ee8178845436d3644e 2023-04-24 18:56:20,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/51985c92aebc458b9d346e599f2798f2 2023-04-24 18:56:20,362 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:20,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:20,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362640363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362640367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:20,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:20,372 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:20,372 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:20,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,374 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:20,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:20,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:20,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,375 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,380 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cf7847[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,381 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,381 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,383 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,383 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,384 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,386 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,386 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,386 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,388 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,388 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,390 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,393 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,393 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,392 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,393 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,392 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:20,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362640468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362640470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,493 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:20,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:20,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:20,496 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:20,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:20,496 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:20,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:20,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:20,499 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:20,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:20,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:20,500 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,502 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,502 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321a5b63[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,505 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,505 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,506 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,508 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,508 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,508 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,508 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,510 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,510 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,512 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,512 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,512 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,513 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,514 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,517 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,518 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:20,517 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,518 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,518 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,618 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:20,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:20,622 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:20,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:20,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:20,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,623 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,624 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,624 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:20,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:20,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:20,626 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,626 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:20,626 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:20,626 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,628 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704c6c51[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,636 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,636 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,637 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,640 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,640 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,640 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,642 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,642 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,644 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:20,646 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,646 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,646 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,647 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,646 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362640671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:20,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362640674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,747 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:20,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:20,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:20,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:20,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:20,750 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:20,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:20,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:20,751 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,751 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,753 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,753 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,753 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:20,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:20,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:20,757 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,757 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:20,757 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:20,757 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,758 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e286a37[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,760 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,761 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,762 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,762 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,763 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,765 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,765 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,765 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/85bfa33532494e4ea1aefa7ee5fd93f0 2023-04-24 18:56:20,769 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,769 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,769 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,770 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,772 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,775 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,776 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,780 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,780 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,780 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,780 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/85946e3271c443f5b9dd80a8efffe6c7 2023-04-24 18:56:20,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/51985c92aebc458b9d346e599f2798f2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2 2023-04-24 18:56:20,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2, entries=200, sequenceid=241, filesize=32.1 K 2023-04-24 18:56:20,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/85bfa33532494e4ea1aefa7ee5fd93f0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0 2023-04-24 18:56:20,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0, entries=150, sequenceid=241, filesize=11.8 K 2023-04-24 18:56:20,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/85946e3271c443f5b9dd80a8efffe6c7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7 2023-04-24 18:56:20,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7, entries=150, sequenceid=241, filesize=11.8 K 2023-04-24 18:56:20,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a5268f22ea5dbabf8eac37708fa245e in 468ms, sequenceid=241, compaction requested=true 2023-04-24 18:56:20,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:20,805 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:20,805 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:20,805 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:20,805 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:20,805 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:20,805 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:20,806 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:20,806 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,806 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8e240e57c0d94418a12dadf5d9cb8cb7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=83.0 K 2023-04-24 18:56:20,806 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:20,806 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:20,806 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,806 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/02e7a3ec13b941cb9d64e8dda84b6788, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.6 K 2023-04-24 18:56:20,806 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8e240e57c0d94418a12dadf5d9cb8cb7, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:20,807 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 8aec85c73fb44d8f8fc0bc68b93f62c7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682362578086 2023-04-24 18:56:20,807 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 02e7a3ec13b941cb9d64e8dda84b6788, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:20,807 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 51985c92aebc458b9d346e599f2798f2, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:20,807 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4227d7a295394c808d88216d67d51eb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682362578086 2023-04-24 18:56:20,808 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 85bfa33532494e4ea1aefa7ee5fd93f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:20,814 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#537 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:20,825 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#538 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:20,832 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/c930f56af83e4c6a94e7715249c1900a as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c930f56af83e4c6a94e7715249c1900a 2023-04-24 18:56:20,836 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into c930f56af83e4c6a94e7715249c1900a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:20,836 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:20,836 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=13, startTime=1682362580805; duration=0sec 2023-04-24 18:56:20,836 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:20,836 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:20,838 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:20,838 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:20,838 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,838 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/220d02ae5b9f437d8a32cd3c5974217f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.6 K 2023-04-24 18:56:20,839 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 220d02ae5b9f437d8a32cd3c5974217f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682362577416 2023-04-24 18:56:20,839 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 7c8cc1900cec46b9a30468ee6027c369, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1682362578086 2023-04-24 18:56:20,839 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 85946e3271c443f5b9dd80a8efffe6c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:20,845 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#539 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:20,854 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/efd57ebed5b94ea294fbdfd3bbcc5fb7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/efd57ebed5b94ea294fbdfd3bbcc5fb7 2023-04-24 18:56:20,859 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into efd57ebed5b94ea294fbdfd3bbcc5fb7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:20,859 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:20,859 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=13, startTime=1682362580805; duration=0sec 2023-04-24 18:56:20,859 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:20,880 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:20,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:20,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:20,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:20,884 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:20,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:20,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:20,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,884 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,886 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,886 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:20,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:20,888 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:20,905 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d7de065847684ebe98ca2ebe5c2cc587 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7de065847684ebe98ca2ebe5c2cc587 2023-04-24 18:56:20,918 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5ab5ab82c08f4195a9898af7861bbdec 2023-04-24 18:56:20,929 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/9ede92af367e40438b5553e7300ed6c9 2023-04-24 18:56:20,939 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/1d9cd0504c674add8ab6f0d63d0ee757 2023-04-24 18:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5ab5ab82c08f4195a9898af7861bbdec as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec 2023-04-24 18:56:20,948 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec, entries=150, sequenceid=261, filesize=25.4 K 2023-04-24 18:56:20,948 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/9ede92af367e40438b5553e7300ed6c9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9 2023-04-24 18:56:20,951 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9, entries=150, sequenceid=261, filesize=11.9 K 2023-04-24 18:56:20,952 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/1d9cd0504c674add8ab6f0d63d0ee757 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757 2023-04-24 18:56:20,957 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757, entries=150, sequenceid=261, filesize=11.9 K 2023-04-24 18:56:20,958 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 1a5268f22ea5dbabf8eac37708fa245e in 70ms, sequenceid=261, compaction requested=false 2023-04-24 18:56:20,958 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:20,958 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:20,960 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176c3cf5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:20,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:20,962 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,964 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:20,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,966 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:20,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:20,969 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:20,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:20,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:20,969 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:20,969 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:20,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,972 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:20,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:20,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:20,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:20,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:20,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:20,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:20,976 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:20,976 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:20,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:20,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:21,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:21,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:21,031 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424f527a52dff3f4a0491c63c5af1f39763 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f527a52dff3f4a0491c63c5af1f39763 2023-04-24 18:56:21,037 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/b803ceb6bb2140e1b203f1fd8b1adda6 2023-04-24 18:56:21,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/eb1d6d1f8be6424682f7ce2e146015f2 2023-04-24 18:56:21,059 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/567505c2e59a4990876544d9e244f09f 2023-04-24 18:56:21,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/b803ceb6bb2140e1b203f1fd8b1adda6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6 2023-04-24 18:56:21,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6, entries=250, sequenceid=271, filesize=39.2 K 2023-04-24 18:56:21,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/eb1d6d1f8be6424682f7ce2e146015f2 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2 2023-04-24 18:56:21,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581052, max timestamp=1682362581066Num uniques 859; 2023-04-24 18:56:21,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581052, max timestamp=1682362581066Num uniques 859; 2023-04-24 18:56:21,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581052, max timestamp=1682362581066Num uniques 859; 2023-04-24 18:56:21,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2, entries=150, sequenceid=271, filesize=11.9 K 2023-04-24 18:56:21,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/567505c2e59a4990876544d9e244f09f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f 2023-04-24 18:56:21,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f, entries=150, sequenceid=271, filesize=11.9 K 2023-04-24 18:56:21,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=46.96 KB/48090 for 1a5268f22ea5dbabf8eac37708fa245e in 57ms, sequenceid=271, compaction requested=true 2023-04-24 18:56:21,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:21,076 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:21,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:21,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:56:21,076 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-24 18:56:21,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:21,077 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-24 18:56:21,077 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-24 18:56:21,077 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. because compaction request was cancelled 2023-04-24 18:56:21,077 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:21,078 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:21,078 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:21,078 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:21,078 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c930f56af83e4c6a94e7715249c1900a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.9 K 2023-04-24 18:56:21,079 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c930f56af83e4c6a94e7715249c1900a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:21,079 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 9ede92af367e40438b5553e7300ed6c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682362580349 2023-04-24 18:56:21,080 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting eb1d6d1f8be6424682f7ce2e146015f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682362581014 2023-04-24 18:56:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:21,080 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:21,080 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,080 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:21,080 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:21,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:21,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-24 18:56:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,081 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:21,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:21,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,088 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:21,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:21,092 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,099 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:21,101 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cdac38[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:21,103 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:21,103 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,104 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:21,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,105 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:21,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:21,109 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,109 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:21,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:21,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:21,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:21,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:21,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:21,114 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ef1692979e0745c4a5ddeb5633503e6a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ef1692979e0745c4a5ddeb5633503e6a 2023-04-24 18:56:21,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,116 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,117 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:21,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,121 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:21,121 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:21,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581110, max timestamp=1682362581120Num uniques 859; 2023-04-24 18:56:21,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:21,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581110, max timestamp=1682362581120Num uniques 859; 2023-04-24 18:56:21,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362581110, max timestamp=1682362581120Num uniques 859; 2023-04-24 18:56:21,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:21,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362641151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362641155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,221 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:21,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:21,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:21,225 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:21,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:21,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:21,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,225 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,226 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,226 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:21,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:21,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:21,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:21,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:21,228 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:21,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f07874[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:21,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:21,238 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,239 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:21,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:21,239 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:21,241 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:21,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:21,241 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:21,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,249 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:21,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:21,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,249 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:21,249 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:21,249 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:21,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362641255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362641259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,349 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:21,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:21,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:21,353 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:21,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:21,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:21,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,355 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,355 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,356 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:21,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:21,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:21,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:21,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:21,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:21,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:21,360 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:21,362 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7914cf0f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:21,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:21,364 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,365 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:21,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:21,365 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:21,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,365 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:21,368 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:21,370 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:21,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:21,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:21,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:21,371 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:21,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,374 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,375 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:21,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:21,378 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,378 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,379 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,379 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,379 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,379 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:21,379 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:21,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362641461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362641463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,479 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:21,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:21,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:21,484 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:21,484 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,485 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,486 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:21,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:21,488 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:21,489 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:21,490 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:21,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f69df4c[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:21,493 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:21,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:21,494 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:21,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:21,497 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:21,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:21,500 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:21,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:21,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:21,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:21,500 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:21,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,501 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,502 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,508 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,508 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:21,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,509 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:21,508 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:21,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:21,509 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:21,509 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:21,509 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,517 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/78c8378082c0435ebafb5295f8bd23ce as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/78c8378082c0435ebafb5295f8bd23ce 2023-04-24 18:56:21,522 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into 78c8378082c0435ebafb5295f8bd23ce(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:21,522 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,522 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=13, startTime=1682362581076; duration=0sec 2023-04-24 18:56:21,523 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:21,523 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:21,524 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:21,524 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:21,524 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,524 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/efd57ebed5b94ea294fbdfd3bbcc5fb7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=35.9 K 2023-04-24 18:56:21,525 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=297, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/53bed13f0c06464a99ad7a3b2ca40c55 2023-04-24 18:56:21,525 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting efd57ebed5b94ea294fbdfd3bbcc5fb7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:21,525 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 1d9cd0504c674add8ab6f0d63d0ee757, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1682362580349 2023-04-24 18:56:21,526 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 567505c2e59a4990876544d9e244f09f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682362581014 2023-04-24 18:56:21,533 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#549 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:21,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/16ee025fff4a4569868e27f577b0d7af 2023-04-24 18:56:21,552 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/c5a5664c4a9d494fb5453f40e1b7704e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c5a5664c4a9d494fb5453f40e1b7704e 2023-04-24 18:56:21,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/72caba6f6a7e4168882afbb7a8004718 2023-04-24 18:56:21,557 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into c5a5664c4a9d494fb5453f40e1b7704e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:21,557 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,557 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=13, startTime=1682362581076; duration=0sec 2023-04-24 18:56:21,557 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:21,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/53bed13f0c06464a99ad7a3b2ca40c55 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55 2023-04-24 18:56:21,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55, entries=150, sequenceid=297, filesize=25.4 K 2023-04-24 18:56:21,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/16ee025fff4a4569868e27f577b0d7af as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af 2023-04-24 18:56:21,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af, entries=150, sequenceid=297, filesize=11.9 K 2023-04-24 18:56:21,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/72caba6f6a7e4168882afbb7a8004718 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718 2023-04-24 18:56:21,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718, entries=150, sequenceid=297, filesize=11.9 K 2023-04-24 18:56:21,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 1a5268f22ea5dbabf8eac37708fa245e in 499ms, sequenceid=297, compaction requested=true 2023-04-24 18:56:21,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:21,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:21,580 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 3 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:21,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:56:21,581 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:21,581 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction 2023-04-24 18:56:21,581 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,582 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=90.0 K 2023-04-24 18:56:21,582 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 5ab5ab82c08f4195a9898af7861bbdec, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=261 2023-04-24 18:56:21,582 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b803ceb6bb2140e1b203f1fd8b1adda6, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=271 2023-04-24 18:56:21,582 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 53bed13f0c06464a99ad7a3b2ca40c55, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=297 2023-04-24 18:56:21,591 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#551 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:21,611 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:21,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:21,615 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:21,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,622 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:21,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:21,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:21,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:21,624 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:21,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:21,625 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:21,625 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:21,628 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/092a4549e3454006aa8052035a1cab93 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/092a4549e3454006aa8052035a1cab93 2023-04-24 18:56:21,634 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 092a4549e3454006aa8052035a1cab93(size=25.5 K), total size for store is 108.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:21,634 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,634 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=10, startTime=1682362581580; duration=0sec 2023-04-24 18:56:21,634 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 18:56:21,634 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-24 18:56:21,635 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-24 18:56:21,636 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-24 18:56:21,636 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. because compaction request was cancelled 2023-04-24 18:56:21,636 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-24 18:56:21,643 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-24 18:56:21,643 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-24 18:56:21,643 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. because compaction request was cancelled 2023-04-24 18:56:21,663 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5c236b25442947d5916454a55d4b7499 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5c236b25442947d5916454a55d4b7499 2023-04-24 18:56:21,668 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 5c236b25442947d5916454a55d4b7499(size=25.7 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:21,668 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:21,668 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=13, startTime=1682362580805; duration=0sec 2023-04-24 18:56:21,668 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:21,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:21,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362641778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362641779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362641882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:21,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:21,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362641882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,057 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042480fa83a55b4c4dc987ef6a8fbddf0ed3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042480fa83a55b4c4dc987ef6a8fbddf0ed3 2023-04-24 18:56:22,062 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/d533fb2041224cf78158576e0bef4ddc 2023-04-24 18:56:22,073 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/768395f4645b4f91990cf5c3b630ff3e 2023-04-24 18:56:22,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362642084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362642086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,094 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/9524faa5fcde4985b16dc0260114b52e 2023-04-24 18:56:22,105 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/d533fb2041224cf78158576e0bef4ddc as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc 2023-04-24 18:56:22,109 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc, entries=150, sequenceid=335, filesize=25.4 K 2023-04-24 18:56:22,109 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/768395f4645b4f91990cf5c3b630ff3e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e 2023-04-24 18:56:22,112 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e, entries=150, sequenceid=335, filesize=11.9 K 2023-04-24 18:56:22,113 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/9524faa5fcde4985b16dc0260114b52e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e 2023-04-24 18:56:22,115 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e, entries=150, sequenceid=335, filesize=11.9 K 2023-04-24 18:56:22,116 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1a5268f22ea5dbabf8eac37708fa245e in 491ms, sequenceid=335, compaction requested=true 2023-04-24 18:56:22,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:22,116 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:22,118 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,118 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:22,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,119 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155f0ef5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:22,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:22,121 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,122 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:22,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:22,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:22,125 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:22,128 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:22,128 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:22,128 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:22,128 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:22,128 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:22,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:22,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,132 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:22,133 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,134 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:22,137 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:22,137 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:22,138 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:22,138 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:22,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:22,238 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:22,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:22,242 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:22,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:22,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:22,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:22,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:22,243 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:22,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:22,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:22,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:22,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:22,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:22,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:22,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:22,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:22,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:22,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:22,245 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:22,246 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:22,246 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:22,262 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424dba28ee64cd841e0a410fddcf0d56b4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dba28ee64cd841e0a410fddcf0d56b4a 2023-04-24 18:56:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:22,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:22,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362642431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362582422, max timestamp=1682362582434Num uniques 859; 2023-04-24 18:56:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:22,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682362582422, max timestamp=1682362582434Num uniques 859; 2023-04-24 18:56:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 18:56:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682362582422, max timestamp=1682362582434Num uniques 859; 2023-04-24 18:56:22,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-24 18:56:22,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362642453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362642454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362642456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362642456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362642536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362642557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362642558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362642560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362642560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,668 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/2340becfc1b74b41a401359fe60171d3 2023-04-24 18:56:22,678 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/936a40ffec0549eca887445ecc0d89fb 2023-04-24 18:56:22,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362642739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362642761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362642762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362642762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362642762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362643043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362643065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362643067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362643067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362643069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,090 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/fc304df384514c77a5c90a62f2638768 2023-04-24 18:56:23,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/2340becfc1b74b41a401359fe60171d3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3 2023-04-24 18:56:23,097 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3, entries=150, sequenceid=348, filesize=25.4 K 2023-04-24 18:56:23,098 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/936a40ffec0549eca887445ecc0d89fb as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb 2023-04-24 18:56:23,101 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb, entries=150, sequenceid=348, filesize=11.9 K 2023-04-24 18:56:23,102 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/fc304df384514c77a5c90a62f2638768 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768 2023-04-24 18:56:23,105 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768, entries=150, sequenceid=348, filesize=11.9 K 2023-04-24 18:56:23,105 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127835 for 1a5268f22ea5dbabf8eac37708fa245e in 859ms, sequenceid=348, compaction requested=true 2023-04-24 18:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,108 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,108 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:23,108 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@720be7f2[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:23,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:23,111 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,113 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:23,113 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,113 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:23,115 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:23,119 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:23,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:23,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:23,119 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:23,119 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:23,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,121 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:23,122 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,123 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:23,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:23,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:23,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:23,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:23,126 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:23,126 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:23,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:23,127 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,127 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:23,127 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:23,227 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:23,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:23,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:23,231 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:23,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:23,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:23,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:23,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,231 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:23,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:23,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:23,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,236 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:23,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:23,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:23,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:23,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:23,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:23,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:23,238 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=124.84 KB heapSize=354.38 KB 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:23,238 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:23,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:23,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362643571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362643572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362643574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362643574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362643577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,653 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304246ab4ba1123364039ac95cfc0c38fb349 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246ab4ba1123364039ac95cfc0c38fb349 2023-04-24 18:56:23,658 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/f6b811e5427b415ca41e8ea94f0f6057 2023-04-24 18:56:23,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362643676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362643678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362643882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:23,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362643883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,068 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0840ee96b05b4fe0a036d32fccac6f33 2023-04-24 18:56:24,080 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/76149e6f1f354457818f74fc2e339f92 2023-04-24 18:56:24,084 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/f6b811e5427b415ca41e8ea94f0f6057 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057 2023-04-24 18:56:24,087 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057, entries=150, sequenceid=383, filesize=25.4 K 2023-04-24 18:56:24,088 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0840ee96b05b4fe0a036d32fccac6f33 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33 2023-04-24 18:56:24,095 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33, entries=150, sequenceid=383, filesize=11.9 K 2023-04-24 18:56:24,095 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/76149e6f1f354457818f74fc2e339f92 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92 2023-04-24 18:56:24,099 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92, entries=150, sequenceid=383, filesize=11.9 K 2023-04-24 18:56:24,100 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~124.84 KB/127835, heapSize ~354.34 KB/362840, currentSize=53.67 KB/54960 for 1a5268f22ea5dbabf8eac37708fa245e in 862ms, sequenceid=383, compaction requested=true 2023-04-24 18:56:24,100 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:24,100 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,101 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,102 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db8320b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,104 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,105 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,106 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,106 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,111 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,114 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,118 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,118 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,120 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,123 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,123 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,123 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,123 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,124 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:24,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:56:24,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:24,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:24,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:24,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:24,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:24,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:24,200 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424738ef9a8e66b4b13a862c7b1a9650726 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424738ef9a8e66b4b13a862c7b1a9650726 2023-04-24 18:56:24,204 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/4cdf8dd66f4d407db5ab41c3238fa477 2023-04-24 18:56:24,224 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,227 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,227 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,228 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,228 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,229 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362644226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362644227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,233 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,234 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cea0b4f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,235 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,235 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,237 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,237 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,238 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,238 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,241 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,241 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,241 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,242 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,244 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,247 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,247 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,248 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,247 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,248 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362644330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362644330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,348 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,351 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,353 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,353 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,354 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,355 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,357 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f8881[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,359 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,360 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,360 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,360 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,364 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,367 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,367 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,367 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,369 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,369 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,369 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,369 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,370 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,373 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,374 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,374 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,374 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,374 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,374 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,474 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,478 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,478 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,479 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,479 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,480 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,481 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,481 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,482 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,484 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c36db21[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,486 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,487 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,487 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,487 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,489 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,489 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,492 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,492 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,492 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,493 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,494 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,495 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,500 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,499 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,500 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,500 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362644534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362644534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362644578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362644585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362644585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,600 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,604 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,604 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,608 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,608 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,610 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,610 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,611 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,611 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0bae93c10fe8490487ebe1adcd69ca34 2023-04-24 18:56:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40728cc6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,614 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,616 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,619 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,619 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,619 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,622 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,626 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,627 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,627 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,627 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,727 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,731 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,731 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,733 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,733 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,735 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,735 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,737 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfe57f4[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,739 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,740 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,741 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,740 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,741 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,742 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,742 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,745 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,745 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,745 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,745 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,745 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,748 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,749 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,750 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,752 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,752 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,753 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,753 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,753 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362644836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362644838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,853 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,857 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,857 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,858 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,858 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,859 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,863 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,864 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,864 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,865 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168a36a[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,866 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,868 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,869 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,870 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:24,872 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:24,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:24,872 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:24,873 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:24,873 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:24,874 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,874 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,874 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,875 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,876 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,878 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,878 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,878 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:24,878 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,879 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,879 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,879 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,879 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,879 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:24,879 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:24,878 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:24,878 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:24,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:24,979 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:24,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:24,982 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:24,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:24,982 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:24,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,985 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:24,985 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:24,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:24,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:24,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:24,987 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,987 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:24,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:24,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:24,989 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,990 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b9d9fb5[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:24,992 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:24,992 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:24,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,996 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:24,998 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:25,001 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:25,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:25,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:25,002 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:25,002 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:25,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,003 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,004 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,005 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:25,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:25,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:25,008 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:25,008 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,008 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:25,009 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:25,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:25,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/a5dd32c8769f4d5b9d0d894cc459d067 2023-04-24 18:56:25,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/4cdf8dd66f4d407db5ab41c3238fa477 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477 2023-04-24 18:56:25,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477, entries=200, sequenceid=396, filesize=32.3 K 2023-04-24 18:56:25,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0bae93c10fe8490487ebe1adcd69ca34 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34 2023-04-24 18:56:25,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34, entries=150, sequenceid=396, filesize=11.9 K 2023-04-24 18:56:25,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/a5dd32c8769f4d5b9d0d894cc459d067 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067 2023-04-24 18:56:25,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067, entries=150, sequenceid=396, filesize=11.9 K 2023-04-24 18:56:25,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a5268f22ea5dbabf8eac37708fa245e in 849ms, sequenceid=396, compaction requested=true 2023-04-24 18:56:25,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:25,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:25,038 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:25,038 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:25,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:25,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:25,048 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163713 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:25,048 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:25,048 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:25,048 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:25,048 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:25,048 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:25,048 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5c236b25442947d5916454a55d4b7499, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/092a4549e3454006aa8052035a1cab93, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=159.9 K 2023-04-24 18:56:25,048 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/78c8378082c0435ebafb5295f8bd23ce, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=71.9 K 2023-04-24 18:56:25,049 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5c236b25442947d5916454a55d4b7499, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682362579214 2023-04-24 18:56:25,049 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 78c8378082c0435ebafb5295f8bd23ce, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682362581014 2023-04-24 18:56:25,049 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 092a4549e3454006aa8052035a1cab93, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682362581074 2023-04-24 18:56:25,049 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 16ee025fff4a4569868e27f577b0d7af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682362581074 2023-04-24 18:56:25,050 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 768395f4645b4f91990cf5c3b630ff3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682362581143 2023-04-24 18:56:25,050 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting d533fb2041224cf78158576e0bef4ddc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682362581143 2023-04-24 18:56:25,050 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 936a40ffec0549eca887445ecc0d89fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682362581765 2023-04-24 18:56:25,050 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 2340becfc1b74b41a401359fe60171d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682362581765 2023-04-24 18:56:25,050 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0840ee96b05b4fe0a036d32fccac6f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682362582445 2023-04-24 18:56:25,051 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting f6b811e5427b415ca41e8ea94f0f6057, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682362582445 2023-04-24 18:56:25,051 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0bae93c10fe8490487ebe1adcd69ca34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:25,051 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 4cdf8dd66f4d407db5ab41c3238fa477, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:25,065 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#564 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:25,072 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#565 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:25,099 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/e6cca71589da4c44a8da45a93d4b46e3 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/e6cca71589da4c44a8da45a93d4b46e3 2023-04-24 18:56:25,103 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into e6cca71589da4c44a8da45a93d4b46e3(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:25,103 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:25,103 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=10, startTime=1682362585038; duration=0sec 2023-04-24 18:56:25,104 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:25,104 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:25,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:25,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:25,105 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:25,106 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c5a5664c4a9d494fb5453f40e1b7704e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=71.9 K 2023-04-24 18:56:25,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c5a5664c4a9d494fb5453f40e1b7704e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682362581014 2023-04-24 18:56:25,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 72caba6f6a7e4168882afbb7a8004718, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682362581074 2023-04-24 18:56:25,106 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9524faa5fcde4985b16dc0260114b52e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682362581143 2023-04-24 18:56:25,107 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting fc304df384514c77a5c90a62f2638768, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682362581765 2023-04-24 18:56:25,107 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 76149e6f1f354457818f74fc2e339f92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682362582445 2023-04-24 18:56:25,107 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting a5dd32c8769f4d5b9d0d894cc459d067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:25,109 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:25,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:25,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:25,112 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:25,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:25,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:25,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:25,113 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,113 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:25,114 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,114 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:25,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,115 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:25,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:25,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:25,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:25,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:25,117 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:25,117 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:25,122 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#566 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:25,141 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/79483d2f6f714eddb28b362839571eae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/79483d2f6f714eddb28b362839571eae 2023-04-24 18:56:25,142 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042444b13d225cb64313a5e3d7a70a1a973f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444b13d225cb64313a5e3d7a70a1a973f 2023-04-24 18:56:25,148 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into 79483d2f6f714eddb28b362839571eae(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:25,148 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:25,148 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=10, startTime=1682362585039; duration=0sec 2023-04-24 18:56:25,148 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:25,345 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:25,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362645361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362645361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362645465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362645465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,479 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/cea316077a4d48b781ec87f646c9cc04 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/cea316077a4d48b781ec87f646c9cc04 2023-04-24 18:56:25,487 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into cea316077a4d48b781ec87f646c9cc04(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:25,487 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:25,487 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=10, startTime=1682362585038; duration=0sec 2023-04-24 18:56:25,487 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:25,551 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/498187c88c5e4fc2b4fbca252ac11383 2023-04-24 18:56:25,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362645668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362645668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,960 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/79f988814f3d4049b47eaa4fbe446afe 2023-04-24 18:56:25,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362645971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:25,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362645971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,973 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/b9d3ef910b5c46cba7b3714c7c9cdd07 2023-04-24 18:56:25,977 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/498187c88c5e4fc2b4fbca252ac11383 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383 2023-04-24 18:56:25,980 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383, entries=150, sequenceid=416, filesize=25.4 K 2023-04-24 18:56:25,981 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/79f988814f3d4049b47eaa4fbe446afe as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe 2023-04-24 18:56:25,984 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe, entries=150, sequenceid=416, filesize=11.9 K 2023-04-24 18:56:25,984 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/b9d3ef910b5c46cba7b3714c7c9cdd07 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07 2023-04-24 18:56:25,988 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07, entries=150, sequenceid=416, filesize=11.9 K 2023-04-24 18:56:25,988 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 1a5268f22ea5dbabf8eac37708fa245e in 871ms, sequenceid=416, compaction requested=false 2023-04-24 18:56:25,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:25,988 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,989 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:25,989 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:25,991 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fdd34bc[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:25,993 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:25,993 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,994 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:25,995 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,995 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:25,995 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:25,998 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:25,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:25,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:25,998 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:25,998 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:25,999 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,999 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,000 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:26,000 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,001 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,005 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,006 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:26,006 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:26,006 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:26,006 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:26,006 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:26,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:26,106 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:26,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:26,110 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:26,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:26,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:26,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:26,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,110 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,112 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:26,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:26,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,112 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:26,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:26,114 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:26,115 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:26,130 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424cbadc660943c43acb5ae0bd765a12f17 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cbadc660943c43acb5ae0bd765a12f17 2023-04-24 18:56:26,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:26,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362646500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362646500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,536 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=432, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/9756b2dfd1ac44f5a8d2460a87940c36 2023-04-24 18:56:26,544 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/c33488e8dc48436cba00f27f2046eb7f 2023-04-24 18:56:26,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362646589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362646591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362646592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,594 DEBUG [Thread-3762] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:26,595 DEBUG [Thread-3766] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:26,596 DEBUG [Thread-3770] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., hostname=jenkins-hbase4.apache.org,37593,1682362415680, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 18:56:26,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362646604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362646604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362646807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:26,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362646807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,953 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/c2ac771c33bb4c45ac00425573478e5d 2023-04-24 18:56:26,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/9756b2dfd1ac44f5a8d2460a87940c36 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36 2023-04-24 18:56:26,960 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36, entries=150, sequenceid=432, filesize=25.4 K 2023-04-24 18:56:26,961 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/c33488e8dc48436cba00f27f2046eb7f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f 2023-04-24 18:56:26,964 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f, entries=150, sequenceid=432, filesize=11.9 K 2023-04-24 18:56:26,965 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/c2ac771c33bb4c45ac00425573478e5d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d 2023-04-24 18:56:26,968 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d, entries=150, sequenceid=432, filesize=11.9 K 2023-04-24 18:56:26,969 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 1a5268f22ea5dbabf8eac37708fa245e in 855ms, sequenceid=432, compaction requested=true 2023-04-24 18:56:26,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:26,969 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:26,970 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,971 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:26,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:26,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:26,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:26,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41928b44[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:26,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:26,973 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,975 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:26,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,975 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:26,976 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,976 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:26,979 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:26,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:26,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:26,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:26,979 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:26,982 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,982 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:26,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,982 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:26,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,983 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:26,986 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:26,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:26,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:26,986 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:26,988 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:26,988 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:27,088 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:27,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:27,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:27,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:27,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:27,092 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:27,092 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:27,092 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:27,092 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:27,093 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,093 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:27,095 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,095 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:27,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:27,095 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,095 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:27,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:27,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:27,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:27,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:27,096 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:27,097 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:27,097 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:27,097 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:27,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:27,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362647126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362647129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362647231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362647234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34382 deadline: 1682362647435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362647438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,517 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304245e35b125c15c4464becdf442ead9e067 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e35b125c15c4464becdf442ead9e067 2023-04-24 18:56:27,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362647738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:27,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362647743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,924 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/cebda979212947d4bbcc0ee4ee74b6b8 2023-04-24 18:56:27,933 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/c22d3a0ab09c46f2815525f16e89f835 2023-04-24 18:56:27,942 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/4119e7d78ca5433da09537f07d0195a8 2023-04-24 18:56:27,945 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/cebda979212947d4bbcc0ee4ee74b6b8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8 2023-04-24 18:56:27,948 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8, entries=150, sequenceid=451, filesize=25.4 K 2023-04-24 18:56:27,949 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/c22d3a0ab09c46f2815525f16e89f835 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835 2023-04-24 18:56:27,952 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835, entries=150, sequenceid=451, filesize=11.9 K 2023-04-24 18:56:27,957 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/4119e7d78ca5433da09537f07d0195a8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8 2023-04-24 18:56:27,961 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8, entries=150, sequenceid=451, filesize=11.9 K 2023-04-24 18:56:27,961 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a5268f22ea5dbabf8eac37708fa245e in 864ms, sequenceid=451, compaction requested=true 2023-04-24 18:56:27,961 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:27,961 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:27,963 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:27,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:27,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a503ef[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:27,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:27,965 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,967 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:27,967 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:27,968 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,968 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:27,973 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:27,973 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,973 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:27,976 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:27,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:27,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:27,976 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:27,976 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:27,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:27,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,978 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:27,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,979 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,981 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:27,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:27,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:27,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:27,984 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:27,984 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:27,984 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:27,984 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:27,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:27,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,085 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:28,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:28,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:28,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:28,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:28,088 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:28,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:28,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:28,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,088 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,091 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,091 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:28,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:28,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:28,093 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:28,094 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:28,246 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362648273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362648275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362648376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362648378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,506 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424524b5fcc259c40198aabbbf3a11eb6d1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424524b5fcc259c40198aabbbf3a11eb6d1 2023-04-24 18:56:28,513 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=464, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/728dea78086b4bb49d543fb27c587445 2023-04-24 18:56:28,522 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/f74aa6383dcc48a2af6b84449513c4b7 2023-04-24 18:56:28,538 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/a836f42de0064303a87e2b94ee96f28d 2023-04-24 18:56:28,542 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/728dea78086b4bb49d543fb27c587445 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445 2023-04-24 18:56:28,546 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445, entries=150, sequenceid=464, filesize=25.4 K 2023-04-24 18:56:28,547 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/f74aa6383dcc48a2af6b84449513c4b7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7 2023-04-24 18:56:28,553 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7, entries=150, sequenceid=464, filesize=11.9 K 2023-04-24 18:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/a836f42de0064303a87e2b94ee96f28d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d 2023-04-24 18:56:28,557 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d, entries=150, sequenceid=464, filesize=11.9 K 2023-04-24 18:56:28,557 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a5268f22ea5dbabf8eac37708fa245e in 464ms, sequenceid=464, compaction requested=true 2023-04-24 18:56:28,557 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:28,557 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:28,558 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:28,559 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723254d9[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:28,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:28,561 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,562 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:28,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,562 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:28,563 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,563 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:28,565 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:28,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:28,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:28,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:28,565 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:28,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,568 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,569 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,570 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,572 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,572 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:28,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:28,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,573 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:28,573 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:28,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:28,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:28,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362648600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362648601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,673 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:28,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:28,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:28,680 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:28,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:28,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:28,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,680 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,682 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,682 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:28,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:28,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:28,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:28,685 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,686 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:28,686 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:28,686 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:28,687 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,687 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ece6f7[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:28,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:28,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,691 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:28,691 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,692 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:28,693 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:28,695 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:28,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:28,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:28,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:28,695 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:28,697 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,697 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,697 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,697 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,699 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,702 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:28,702 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:28,701 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362648703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362648712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,802 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:28,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:28,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:28,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:28,805 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:28,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:28,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:28,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,805 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,808 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,808 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:28,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:28,810 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,811 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f18f85[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:28,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:28,814 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,815 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:28,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,815 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:28,816 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:28,819 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:28,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:28,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:28,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:28,819 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:28,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,822 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,823 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,828 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:28,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,829 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,829 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:28,829 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:28,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:28,829 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362648905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:28,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362648916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,929 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:28,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:28,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:28,933 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:28,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:28,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:28,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,933 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,934 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,934 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,935 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:28,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:28,936 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:28,939 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c665e2f[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:28,940 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:28,941 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,941 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:28,943 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:28,943 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:28,943 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:28,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,946 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:28,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,948 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,951 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:28,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:28,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:28,951 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:28,951 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:28,951 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:28,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:28,996 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304249b6a9f7f46774c458d7f7f27228de1d6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b6a9f7f46774c458d7f7f27228de1d6 2023-04-24 18:56:29,004 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=484, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/c96c222a36214dd4a5daa4ff1e083e7f 2023-04-24 18:56:29,051 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,055 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,055 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,056 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,056 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,057 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,057 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,058 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,058 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,058 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,059 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,060 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9243e6[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,062 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,063 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,064 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,066 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,069 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,069 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,069 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,073 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,075 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,077 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,077 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,078 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,078 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,078 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,078 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,078 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,078 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,178 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,181 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,181 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,182 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,184 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,184 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,185 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,186 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,186 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,188 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f7adfdd[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,190 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,191 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,192 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,193 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,194 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,194 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,196 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,196 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,196 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,199 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,199 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,199 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,200 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,202 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,208 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,208 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,208 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,208 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:29,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362649211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362649220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,308 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,312 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,312 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,313 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,315 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,315 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,317 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,317 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,317 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,317 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,319 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,319 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72faea10[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,321 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,322 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,323 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,323 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,325 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,325 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,327 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,327 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,327 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,329 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,329 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,331 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,333 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,333 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,334 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,334 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,334 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0cf972703be048cda47396a9ea5a74f0 2023-04-24 18:56:29,434 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,437 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,437 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,438 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,439 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,439 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,441 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,441 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,441 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,441 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,444 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,444 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf34546[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,446 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,449 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,449 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,452 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,454 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,454 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,454 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,456 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,457 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,458 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,461 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,461 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,461 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,461 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,562 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,565 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,565 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,566 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,567 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,567 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,568 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,569 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,569 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,569 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,569 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,571 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3220b64d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,573 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,574 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,574 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,577 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,579 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,579 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,579 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,581 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,581 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,581 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,581 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,582 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,584 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,585 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,585 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,585 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,585 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,685 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,689 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,689 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,690 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,695 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,695 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,696 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,697 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,697 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,697 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,697 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,699 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b8d03e3[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,701 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,702 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,702 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,703 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,703 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,703 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,704 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,707 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,707 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,707 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,710 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,711 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,712 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,715 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,715 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,716 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,716 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362649714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362649724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,816 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,819 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,819 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,820 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,821 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,821 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/2acfa1a5ecf041c29e16f9a6ce22f6f6 2023-04-24 18:56:29,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,822 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:29,823 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:29,825 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,825 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,826 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/c96c222a36214dd4a5daa4ff1e083e7f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f 2023-04-24 18:56:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:29,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e02c0e8[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:29,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:29,827 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f, entries=200, sequenceid=484, filesize=32.3 K 2023-04-24 18:56:29,830 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:29,830 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:29,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0cf972703be048cda47396a9ea5a74f0 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0 2023-04-24 18:56:29,830 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,831 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:29,832 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0, entries=150, sequenceid=484, filesize=11.9 K 2023-04-24 18:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/2acfa1a5ecf041c29e16f9a6ce22f6f6 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6 2023-04-24 18:56:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:29,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:29,835 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:29,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:29,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:29,835 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:29,835 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:29,837 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:29,837 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,837 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:29,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:29,837 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:29,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6, entries=150, sequenceid=484, filesize=11.9 K 2023-04-24 18:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,839 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1a5268f22ea5dbabf8eac37708fa245e in 1257ms, sequenceid=484, compaction requested=true 2023-04-24 18:56:29,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:29,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:29,840 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:29,840 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:29,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:29,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,841 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,842 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:29,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 164066 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:29,842 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:29,842 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:29,842 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,842 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:29,842 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/cea316077a4d48b781ec87f646c9cc04, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=72.1 K 2023-04-24 18:56:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:29,842 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,842 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,842 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:29,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:29,843 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting cea316077a4d48b781ec87f646c9cc04, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:29,843 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/e6cca71589da4c44a8da45a93d4b46e3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=160.2 K 2023-04-24 18:56:29,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:29,843 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 79f988814f3d4049b47eaa4fbe446afe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682362584208 2023-04-24 18:56:29,843 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting e6cca71589da4c44a8da45a93d4b46e3, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:29,843 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c33488e8dc48436cba00f27f2046eb7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682362585354 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 498187c88c5e4fc2b4fbca252ac11383, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682362584208 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c22d3a0ab09c46f2815525f16e89f835, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682362586495 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 9756b2dfd1ac44f5a8d2460a87940c36, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682362585354 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting f74aa6383dcc48a2af6b84449513c4b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682362587124 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cebda979212947d4bbcc0ee4ee74b6b8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682362586495 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0cf972703be048cda47396a9ea5a74f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:29,844 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 728dea78086b4bb49d543fb27c587445, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682362587124 2023-04-24 18:56:29,845 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting c96c222a36214dd4a5daa4ff1e083e7f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:29,854 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#582 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:29,859 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#583 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:29,943 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:29,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:29,946 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:29,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:29,946 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:29,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,947 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,949 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:29,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:29,949 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:29,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:29,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:29,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:29,951 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:29,951 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:29,970 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a4a42e794d9b419da392813adde73387 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a4a42e794d9b419da392813adde73387 2023-04-24 18:56:30,264 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/0db4acb791674cf59cdb69c7849d9b18 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0db4acb791674cf59cdb69c7849d9b18 2023-04-24 18:56:30,268 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into 0db4acb791674cf59cdb69c7849d9b18(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:30,268 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:30,268 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=10, startTime=1682362589840; duration=0sec 2023-04-24 18:56:30,268 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:30,268 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 18:56:30,270 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 18:56:30,271 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:30,271 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:30,271 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/79483d2f6f714eddb28b362839571eae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=72.1 K 2023-04-24 18:56:30,271 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 79483d2f6f714eddb28b362839571eae, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682362583571 2023-04-24 18:56:30,272 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting b9d3ef910b5c46cba7b3714c7c9cdd07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682362584208 2023-04-24 18:56:30,272 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting c2ac771c33bb4c45ac00425573478e5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682362585354 2023-04-24 18:56:30,273 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 4119e7d78ca5433da09537f07d0195a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682362586495 2023-04-24 18:56:30,273 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting a836f42de0064303a87e2b94ee96f28d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682362587124 2023-04-24 18:56:30,273 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 2acfa1a5ecf041c29e16f9a6ce22f6f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:30,283 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#585 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:30,290 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/cacae469d93e4c0dba733a7f2c95c887 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/cacae469d93e4c0dba733a7f2c95c887 2023-04-24 18:56:30,294 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into cacae469d93e4c0dba733a7f2c95c887(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:30,294 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:30,294 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=10, startTime=1682362589840; duration=0sec 2023-04-24 18:56:30,294 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:30,375 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=496, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5005542f009247a2a175df7a6cbd09cd 2023-04-24 18:56:30,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:30,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:30,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362650641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362650642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362650643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,677 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/1064c02d17c3412da2d822b6c669eaab as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1064c02d17c3412da2d822b6c669eaab 2023-04-24 18:56:30,681 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into 1064c02d17c3412da2d822b6c669eaab(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:30,681 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:30,681 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=10, startTime=1682362589840; duration=0sec 2023-04-24 18:56:30,681 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:30,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34412 deadline: 1682362650725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34382 deadline: 1682362650731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362650744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362650745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362650746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,785 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/3c5c9b94bd4c431cb958a6e28579cdc4 2023-04-24 18:56:30,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362650947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362650948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:30,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:30,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362650948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,212 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/d5733c349037446081b037559910b8a7 2023-04-24 18:56:31,216 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/5005542f009247a2a175df7a6cbd09cd as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd 2023-04-24 18:56:31,221 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd, entries=100, sequenceid=496, filesize=18.6 K 2023-04-24 18:56:31,221 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/3c5c9b94bd4c431cb958a6e28579cdc4 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4 2023-04-24 18:56:31,226 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4, entries=100, sequenceid=496, filesize=9.5 K 2023-04-24 18:56:31,226 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/d5733c349037446081b037559910b8a7 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7 2023-04-24 18:56:31,229 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7, entries=100, sequenceid=496, filesize=9.5 K 2023-04-24 18:56:31,230 INFO [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1a5268f22ea5dbabf8eac37708fa245e in 1279ms, sequenceid=496, compaction requested=false 2023-04-24 18:56:31,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:31,230 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,232 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482d6b1b[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:31,234 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:31,234 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,235 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:31,235 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,235 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:31,236 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,237 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:31,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:31,239 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:31,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:31,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:31,239 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:31,239 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:31,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,241 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,242 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,243 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,245 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:31,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:31,245 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,246 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:31,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,246 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:31,246 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:31,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682362590637, max timestamp=1682362591252Num uniques 859; 2023-04-24 18:56:31,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 18:56:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:31,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 18:56:31,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362651270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362651270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362651271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,346 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:31,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:31,350 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:31,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:31,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:31,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,350 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,351 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,351 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,352 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:31,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:31,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:31,354 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,354 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:31,354 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:31,354 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,355 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f883e8d[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:31,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:31,357 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,361 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,361 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,361 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:31,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:31,362 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:31,363 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:31,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:31,366 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:31,366 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:31,366 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:31,366 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:31,366 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:31,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,373 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,375 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362651374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362651375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,376 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362651375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:31,385 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:31,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,385 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,385 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:31,385 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:31,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,485 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:31,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:31,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:31,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:31,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:31,489 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:31,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:31,489 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:31,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,490 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,491 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,491 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,491 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,491 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:31,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:31,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:31,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:31,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,492 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:31,493 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,493 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:31,493 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:31,493 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:31,493 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:31,494 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:31,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c69a884[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:31,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:31,497 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,498 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:31,498 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:31,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,498 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:31,500 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:31,501 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,501 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:31,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:31,504 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:31,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:31,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:31,504 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:31,504 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:31,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,505 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,506 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,507 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,510 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:31,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,510 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,510 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:31,510 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:31,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362651577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37593] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362651577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362651577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,611 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:31,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:31,612 DEBUG [Thread-3781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:59583 2023-04-24 18:56:31,613 DEBUG [Thread-3781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:31,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-24 18:56:31,613 DEBUG [Thread-3773] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:59583 2023-04-24 18:56:31,613 DEBUG [Thread-3773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:31,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 18:56:31,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 18:56:31,614 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 18:56:31,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 18:56:31,615 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 18:56:31,615 DEBUG [Thread-3775] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:59583 2023-04-24 18:56:31,615 DEBUG [Thread-3775] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:31,615 DEBUG [Thread-3779] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:59583 2023-04-24 18:56:31,615 DEBUG [Thread-3779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:31,616 DEBUG [Thread-3777] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:59583 2023-04-24 18:56:31,616 DEBUG [Thread-3777] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:31,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,616 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,618 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,618 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 18:56:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 18:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 18:56:31,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 18:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 18:56:31,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. started... 2023-04-24 18:56:31,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. as already flushing 2023-04-24 18:56:31,620 DEBUG [rs(jenkins-hbase4.apache.org,37593,1682362415680)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 18:56:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 18:56:31,621 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,622 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 18:56:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37593,1682362415680' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 18:56:31,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24443c61[Count = 0] remaining members to acquire global barrier 2023-04-24 18:56:31,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 18:56:31,623 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,624 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,625 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 18:56:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 18:56:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 18:56:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37593,1682362415680' in zk 2023-04-24 18:56:31,626 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 18:56:31,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,626 DEBUG [member: 'jenkins-hbase4.apache.org,37593,1682362415680' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 18:56:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37593,1682362415680': 2023-04-24 18:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37593,1682362415680' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 18:56:31,628 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 18:56:31,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 18:56:31,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 18:56:31,628 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 18:56:31,628 INFO [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 18:56:31,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,630 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 18:56:31,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,630 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 18:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 18:56:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,631 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 18:56:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 18:56:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 18:56:31,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 18:56:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 18:56:31,633 DEBUG [(jenkins-hbase4.apache.org,45309,1682362414518)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 18:56:31,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 18:56:31,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 18:56:31,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 18:56:31,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 18:56:31,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 18:56:31,633 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 18:56:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 18:56:31,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 18:56:31,634 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 18:56:31,634 DEBUG [Thread-3772] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 18:56:31,634 DEBUG [Thread-3772] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 18:56:31,673 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424bd40c1e392d44519ae76ab45a4afc521 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd40c1e392d44519ae76ab45a4afc521 2023-04-24 18:56:31,677 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=520, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/cc3acbc3942d4f0d9700dd3f4668d2b9 2023-04-24 18:56:31,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/32c86579194f41d9902ccfb80a7f88ae 2023-04-24 18:56:31,734 DEBUG [Thread-3772] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 18:56:31,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 18:56:31,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34356 deadline: 1682362651879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34354 deadline: 1682362651880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:31,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 18:56:31,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34338 deadline: 1682362651881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:32,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/634459ecea8345b38b2859ee7d736f8f 2023-04-24 18:56:32,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/cc3acbc3942d4f0d9700dd3f4668d2b9 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9 2023-04-24 18:56:32,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9, entries=150, sequenceid=520, filesize=25.4 K 2023-04-24 18:56:32,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/32c86579194f41d9902ccfb80a7f88ae as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae 2023-04-24 18:56:32,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae, entries=150, sequenceid=520, filesize=11.9 K 2023-04-24 18:56:32,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/634459ecea8345b38b2859ee7d736f8f as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f 2023-04-24 18:56:32,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f, entries=150, sequenceid=520, filesize=11.9 K 2023-04-24 18:56:32,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=87.22 KB/89310 for 1a5268f22ea5dbabf8eac37708fa245e in 851ms, sequenceid=520, compaction requested=true 2023-04-24 18:56:32,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:32,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:32,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:32,104 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:32,104 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:32,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 72044 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/A is initiating minor compaction (all files) 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:32,105 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/A in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/B is initiating minor compaction (all files) 2023-04-24 18:56:32,105 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1064c02d17c3412da2d822b6c669eaab, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=70.4 K 2023-04-24 18:56:32,105 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/B in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:32,105 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0db4acb791674cf59cdb69c7849d9b18, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=34.2 K 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 1064c02d17c3412da2d822b6c669eaab, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 0db4acb791674cf59cdb69c7849d9b18, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting 5005542f009247a2a175df7a6cbd09cd, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682362588595 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 3c5c9b94bd4c431cb958a6e28579cdc4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682362588595 2023-04-24 18:56:32,105 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] compactions.Compactor(207): Compacting cc3acbc3942d4f0d9700dd3f4668d2b9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682362590637 2023-04-24 18:56:32,106 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 32c86579194f41d9902ccfb80a7f88ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682362590637 2023-04-24 18:56:32,110 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#B#compaction#592 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:32,112 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#A#compaction#591 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:32,117 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/f3bd0ae0ead648b2971599a800bae0a5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f3bd0ae0ead648b2971599a800bae0a5 2023-04-24 18:56:32,121 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/B of 1a5268f22ea5dbabf8eac37708fa245e into f3bd0ae0ead648b2971599a800bae0a5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:32,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:32,122 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/B, priority=13, startTime=1682362592104; duration=0sec 2023-04-24 18:56:32,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 18:56:32,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 18:56:32,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 18:56:32,122 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1912): 1a5268f22ea5dbabf8eac37708fa245e/C is initiating minor compaction (all files) 2023-04-24 18:56:32,122 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a5268f22ea5dbabf8eac37708fa245e/C in TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:32,123 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/cacae469d93e4c0dba733a7f2c95c887, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f] into tmpdir=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp, totalSize=34.2 K 2023-04-24 18:56:32,123 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting cacae469d93e4c0dba733a7f2c95c887, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682362588262 2023-04-24 18:56:32,123 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting d5733c349037446081b037559910b8a7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682362588595 2023-04-24 18:56:32,123 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] compactions.Compactor(207): Compacting 634459ecea8345b38b2859ee7d736f8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682362590637 2023-04-24 18:56:32,127 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a5268f22ea5dbabf8eac37708fa245e#C#compaction#593 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 18:56:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37593] regionserver.HRegion(9158): Flush requested on 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:32,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:32,383 DEBUG [Thread-3770] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:59583 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:32,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:32,383 DEBUG [Thread-3770] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,387 DEBUG [Thread-3766] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:59583 2023-04-24 18:56:32,387 DEBUG [Thread-3766] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,392 DEBUG [Thread-3762] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:59583 2023-04-24 18:56:32,392 DEBUG [Thread-3762] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,399 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424fe56c2af616a48829b0c0dca6c9a9a43 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fe56c2af616a48829b0c0dca6c9a9a43 2023-04-24 18:56:32,527 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/db069d0f492946908428a687b11b99ba as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/db069d0f492946908428a687b11b99ba 2023-04-24 18:56:32,530 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/A of 1a5268f22ea5dbabf8eac37708fa245e into db069d0f492946908428a687b11b99ba(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:32,530 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:32,530 INFO [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/A, priority=13, startTime=1682362592104; duration=0sec 2023-04-24 18:56:32,530 DEBUG [RS:0;jenkins-hbase4:37593-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:32,534 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/6551617b867b4b98a6eb9479be302dd8 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/6551617b867b4b98a6eb9479be302dd8 2023-04-24 18:56:32,537 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1a5268f22ea5dbabf8eac37708fa245e/C of 1a5268f22ea5dbabf8eac37708fa245e into 6551617b867b4b98a6eb9479be302dd8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 18:56:32,537 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:32,537 INFO [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e., storeName=1a5268f22ea5dbabf8eac37708fa245e/C, priority=13, startTime=1682362592104; duration=0sec 2023-04-24 18:56:32,538 DEBUG [RS:0;jenkins-hbase4:37593-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 18:56:32,741 DEBUG [Thread-3768] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:59583 2023-04-24 18:56:32,741 DEBUG [Thread-3768] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,749 DEBUG [Thread-3764] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:59583 2023-04-24 18:56:32,749 DEBUG [Thread-3764] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 31 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4723 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4640 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4748 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4760 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(398): read 4680 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 18:56:32,749 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:56:32,749 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:59583 2023-04-24 18:56:32,750 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:32,750 INFO [Listener at localhost/43631] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 18:56:32,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-24 18:56:32,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:32,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:32,753 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362592753"}]},"ts":"1682362592753"} 2023-04-24 18:56:32,754 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 18:56:32,760 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 18:56:32,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, UNASSIGN}] 2023-04-24 18:56:32,761 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, UNASSIGN 2023-04-24 18:56:32,762 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:32,762 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362592762"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682362592762"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682362592762"}]},"ts":"1682362592762"} 2023-04-24 18:56:32,763 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680}] 2023-04-24 18:56:32,807 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=538, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/6650a887f6b94cbfac6ba3155505440e 2023-04-24 18:56:32,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:32,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:32,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1a5268f22ea5dbabf8eac37708fa245e, disabling compactions & flushes 2023-04-24 18:56:32,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:33,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:33,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/bb0ef48c90244e0c8b46a47271f3f0d5 2023-04-24 18:56:33,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:33,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/862065d9137d4e37805487356d334c0e 2023-04-24 18:56:33,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/6650a887f6b94cbfac6ba3155505440e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/6650a887f6b94cbfac6ba3155505440e 2023-04-24 18:56:33,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/6650a887f6b94cbfac6ba3155505440e, entries=150, sequenceid=538, filesize=25.4 K 2023-04-24 18:56:33,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/bb0ef48c90244e0c8b46a47271f3f0d5 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/bb0ef48c90244e0c8b46a47271f3f0d5 2023-04-24 18:56:33,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/bb0ef48c90244e0c8b46a47271f3f0d5, entries=150, sequenceid=538, filesize=11.9 K 2023-04-24 18:56:33,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/862065d9137d4e37805487356d334c0e as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/862065d9137d4e37805487356d334c0e 2023-04-24 18:56:33,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/862065d9137d4e37805487356d334c0e, entries=150, sequenceid=538, filesize=11.9 K 2023-04-24 18:56:33,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=26.84 KB/27480 for 1a5268f22ea5dbabf8eac37708fa245e in 1254ms, sequenceid=538, compaction requested=false 2023-04-24 18:56:33,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:33,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. after waiting 0 ms 2023-04-24 18:56:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:33,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1a5268f22ea5dbabf8eac37708fa245e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 18:56:33,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=A 2023-04-24 18:56:33,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:33,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=B 2023-04-24 18:56:33,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:33,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a5268f22ea5dbabf8eac37708fa245e, store=C 2023-04-24 18:56:33,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 18:56:33,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:34,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ddecc5e46f4747d1bc9cdd0fd82e4911 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ddecc5e46f4747d1bc9cdd0fd82e4911 2023-04-24 18:56:34,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=547, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/77c7a74f02cb44d3bd34fdbb08a4d189 2023-04-24 18:56:34,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/88b3abb6c8e34a9185d007cbcafa0a6d 2023-04-24 18:56:34,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/5075349587f04b97a4892fbf3f733086 2023-04-24 18:56:34,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/A/77c7a74f02cb44d3bd34fdbb08a4d189 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/77c7a74f02cb44d3bd34fdbb08a4d189 2023-04-24 18:56:34,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/77c7a74f02cb44d3bd34fdbb08a4d189, entries=100, sequenceid=547, filesize=18.6 K 2023-04-24 18:56:34,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/B/88b3abb6c8e34a9185d007cbcafa0a6d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/88b3abb6c8e34a9185d007cbcafa0a6d 2023-04-24 18:56:34,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/88b3abb6c8e34a9185d007cbcafa0a6d, entries=100, sequenceid=547, filesize=9.5 K 2023-04-24 18:56:34,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/.tmp/C/5075349587f04b97a4892fbf3f733086 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5075349587f04b97a4892fbf3f733086 2023-04-24 18:56:34,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5075349587f04b97a4892fbf3f733086, entries=100, sequenceid=547, filesize=9.5 K 2023-04-24 18:56:34,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1a5268f22ea5dbabf8eac37708fa245e in 842ms, sequenceid=547, compaction requested=true 2023-04-24 18:56:34,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/883e8cb986624d999a9a913d9b7f4fc5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8e240e57c0d94418a12dadf5d9cb8cb7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5c236b25442947d5916454a55d4b7499, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/092a4549e3454006aa8052035a1cab93, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/e6cca71589da4c44a8da45a93d4b46e3, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1064c02d17c3412da2d822b6c669eaab, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9] to archive 2023-04-24 18:56:34,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:34,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/fac8e019a35a48a981329ea5a97ce9af 2023-04-24 18:56:34,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8d7ecf1404204d83aa85e7d78b3b479b 2023-04-24 18:56:34,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/97fa5e14ca00475192cd8c6f6fe781a6 2023-04-24 18:56:34,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/883e8cb986624d999a9a913d9b7f4fc5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/883e8cb986624d999a9a913d9b7f4fc5 2023-04-24 18:56:34,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/60538da35a0b401b88159de3cf995e43 2023-04-24 18:56:34,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/06e0b4f825074d52a1971f6d9753fe85 2023-04-24 18:56:34,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1f9e01f1551e446d9055a5b99b696564 2023-04-24 18:56:34,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5aa3c4d8a2bf4b38b65ffb8b38f4a98f 2023-04-24 18:56:34,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/a1e05512edcd4e6f9b802671df047a9f 2023-04-24 18:56:34,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8e240e57c0d94418a12dadf5d9cb8cb7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8e240e57c0d94418a12dadf5d9cb8cb7 2023-04-24 18:56:34,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/ad4f643770284967b10801404cfd328d 2023-04-24 18:56:34,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/8aec85c73fb44d8f8fc0bc68b93f62c7 2023-04-24 18:56:34,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/51985c92aebc458b9d346e599f2798f2 2023-04-24 18:56:34,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5c236b25442947d5916454a55d4b7499 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5c236b25442947d5916454a55d4b7499 2023-04-24 18:56:34,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5ab5ab82c08f4195a9898af7861bbdec 2023-04-24 18:56:34,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/b803ceb6bb2140e1b203f1fd8b1adda6 2023-04-24 18:56:34,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/092a4549e3454006aa8052035a1cab93 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/092a4549e3454006aa8052035a1cab93 2023-04-24 18:56:34,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/53bed13f0c06464a99ad7a3b2ca40c55 2023-04-24 18:56:34,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/d533fb2041224cf78158576e0bef4ddc 2023-04-24 18:56:34,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/2340becfc1b74b41a401359fe60171d3 2023-04-24 18:56:34,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/f6b811e5427b415ca41e8ea94f0f6057 2023-04-24 18:56:34,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/4cdf8dd66f4d407db5ab41c3238fa477 2023-04-24 18:56:34,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/e6cca71589da4c44a8da45a93d4b46e3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/e6cca71589da4c44a8da45a93d4b46e3 2023-04-24 18:56:34,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/498187c88c5e4fc2b4fbca252ac11383 2023-04-24 18:56:34,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/9756b2dfd1ac44f5a8d2460a87940c36 2023-04-24 18:56:34,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cebda979212947d4bbcc0ee4ee74b6b8 2023-04-24 18:56:34,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/728dea78086b4bb49d543fb27c587445 2023-04-24 18:56:34,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/c96c222a36214dd4a5daa4ff1e083e7f 2023-04-24 18:56:34,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1064c02d17c3412da2d822b6c669eaab to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/1064c02d17c3412da2d822b6c669eaab 2023-04-24 18:56:34,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/5005542f009247a2a175df7a6cbd09cd 2023-04-24 18:56:34,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/cc3acbc3942d4f0d9700dd3f4668d2b9 2023-04-24 18:56:34,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/26fc198e156745e196c6e1c77b6be391, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d64e2b514ca40f0a1ad0961a3ad108e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/02e7a3ec13b941cb9d64e8dda84b6788, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c930f56af83e4c6a94e7715249c1900a, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/78c8378082c0435ebafb5295f8bd23ce, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/cea316077a4d48b781ec87f646c9cc04, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0db4acb791674cf59cdb69c7849d9b18, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae] to archive 2023-04-24 18:56:34,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:34,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d935bbf93c740979760296f00417c0f 2023-04-24 18:56:34,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/2a1bbc566648429d9b2dc94c6554a42d 2023-04-24 18:56:34,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/054fad0f1e4f43f7a51f4b0db59b0ece 2023-04-24 18:56:34,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/26fc198e156745e196c6e1c77b6be391 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/26fc198e156745e196c6e1c77b6be391 2023-04-24 18:56:34,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/ead48e4519414fc6b5492c6c6abcac71 2023-04-24 18:56:34,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/b97ca1d4e2b445e0acc1020740bdb99a 2023-04-24 18:56:34,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d64e2b514ca40f0a1ad0961a3ad108e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4d64e2b514ca40f0a1ad0961a3ad108e 2023-04-24 18:56:34,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/8c2144dd041e4da8861dde35de94fd51 2023-04-24 18:56:34,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/56f77a1e96a343f9a57edc237d49a976 2023-04-24 18:56:34,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/02e7a3ec13b941cb9d64e8dda84b6788 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/02e7a3ec13b941cb9d64e8dda84b6788 2023-04-24 18:56:34,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/519048f9949540e68033580a8de263d5 2023-04-24 18:56:34,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/4227d7a295394c808d88216d67d51eb6 2023-04-24 18:56:34,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c930f56af83e4c6a94e7715249c1900a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c930f56af83e4c6a94e7715249c1900a 2023-04-24 18:56:34,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/85bfa33532494e4ea1aefa7ee5fd93f0 2023-04-24 18:56:34,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/9ede92af367e40438b5553e7300ed6c9 2023-04-24 18:56:34,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/78c8378082c0435ebafb5295f8bd23ce to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/78c8378082c0435ebafb5295f8bd23ce 2023-04-24 18:56:34,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/eb1d6d1f8be6424682f7ce2e146015f2 2023-04-24 18:56:34,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/16ee025fff4a4569868e27f577b0d7af 2023-04-24 18:56:34,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/768395f4645b4f91990cf5c3b630ff3e 2023-04-24 18:56:34,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/936a40ffec0549eca887445ecc0d89fb 2023-04-24 18:56:34,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0840ee96b05b4fe0a036d32fccac6f33 2023-04-24 18:56:34,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/cea316077a4d48b781ec87f646c9cc04 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/cea316077a4d48b781ec87f646c9cc04 2023-04-24 18:56:34,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0bae93c10fe8490487ebe1adcd69ca34 2023-04-24 18:56:34,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/79f988814f3d4049b47eaa4fbe446afe 2023-04-24 18:56:34,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c33488e8dc48436cba00f27f2046eb7f 2023-04-24 18:56:34,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/c22d3a0ab09c46f2815525f16e89f835 2023-04-24 18:56:34,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f74aa6383dcc48a2af6b84449513c4b7 2023-04-24 18:56:34,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0db4acb791674cf59cdb69c7849d9b18 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0db4acb791674cf59cdb69c7849d9b18 2023-04-24 18:56:34,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/0cf972703be048cda47396a9ea5a74f0 2023-04-24 18:56:34,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/3c5c9b94bd4c431cb958a6e28579cdc4 2023-04-24 18:56:34,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/32c86579194f41d9902ccfb80a7f88ae 2023-04-24 18:56:34,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/e63348c6358b4288b546978a7a04ed34, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d66d1a769544e4f93abc7b990659dc4, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/220d02ae5b9f437d8a32cd3c5974217f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/efd57ebed5b94ea294fbdfd3bbcc5fb7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c5a5664c4a9d494fb5453f40e1b7704e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/79483d2f6f714eddb28b362839571eae, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/cacae469d93e4c0dba733a7f2c95c887, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f] to archive 2023-04-24 18:56:34,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 18:56:34,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/556f290a8b284e8f88401433dbdaee13 2023-04-24 18:56:34,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/bd92e09724d943d3a4c7910844772984 2023-04-24 18:56:34,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/38f097910d4b4447b70e800431122aa4 2023-04-24 18:56:34,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/e63348c6358b4288b546978a7a04ed34 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/e63348c6358b4288b546978a7a04ed34 2023-04-24 18:56:34,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/8483b0ed40614c6aa6e7c37b252925ef 2023-04-24 18:56:34,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d50e3d8488e4764addb639560d5afe4 2023-04-24 18:56:34,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d66d1a769544e4f93abc7b990659dc4 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3d66d1a769544e4f93abc7b990659dc4 2023-04-24 18:56:34,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/3e9e77ef48bb40c28919677e00b86c7e 2023-04-24 18:56:34,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/65f515c6f037423fba2be8a5965fff13 2023-04-24 18:56:34,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/220d02ae5b9f437d8a32cd3c5974217f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/220d02ae5b9f437d8a32cd3c5974217f 2023-04-24 18:56:34,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5e2b0b79199b4670af7349eb253e9bf0 2023-04-24 18:56:34,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/7c8cc1900cec46b9a30468ee6027c369 2023-04-24 18:56:34,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/efd57ebed5b94ea294fbdfd3bbcc5fb7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/efd57ebed5b94ea294fbdfd3bbcc5fb7 2023-04-24 18:56:34,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/85946e3271c443f5b9dd80a8efffe6c7 2023-04-24 18:56:34,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/1d9cd0504c674add8ab6f0d63d0ee757 2023-04-24 18:56:34,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c5a5664c4a9d494fb5453f40e1b7704e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c5a5664c4a9d494fb5453f40e1b7704e 2023-04-24 18:56:34,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/567505c2e59a4990876544d9e244f09f 2023-04-24 18:56:34,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/72caba6f6a7e4168882afbb7a8004718 2023-04-24 18:56:34,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/9524faa5fcde4985b16dc0260114b52e 2023-04-24 18:56:34,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/fc304df384514c77a5c90a62f2638768 2023-04-24 18:56:34,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/76149e6f1f354457818f74fc2e339f92 2023-04-24 18:56:34,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/79483d2f6f714eddb28b362839571eae to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/79483d2f6f714eddb28b362839571eae 2023-04-24 18:56:34,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a5dd32c8769f4d5b9d0d894cc459d067 2023-04-24 18:56:34,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/b9d3ef910b5c46cba7b3714c7c9cdd07 2023-04-24 18:56:34,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/c2ac771c33bb4c45ac00425573478e5d 2023-04-24 18:56:34,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/4119e7d78ca5433da09537f07d0195a8 2023-04-24 18:56:34,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/a836f42de0064303a87e2b94ee96f28d 2023-04-24 18:56:34,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/cacae469d93e4c0dba733a7f2c95c887 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/cacae469d93e4c0dba733a7f2c95c887 2023-04-24 18:56:34,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/2acfa1a5ecf041c29e16f9a6ce22f6f6 2023-04-24 18:56:34,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/d5733c349037446081b037559910b8a7 2023-04-24 18:56:34,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/634459ecea8345b38b2859ee7d736f8f 2023-04-24 18:56:34,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits/550.seqid, newMaxSeqId=550, maxSeqId=4 2023-04-24 18:56:34,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e. 2023-04-24 18:56:34,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1a5268f22ea5dbabf8eac37708fa245e: 2023-04-24 18:56:34,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:34,595 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1a5268f22ea5dbabf8eac37708fa245e, regionState=CLOSED 2023-04-24 18:56:34,595 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682362594595"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682362594595"}]},"ts":"1682362594595"} 2023-04-24 18:56:34,597 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-24 18:56:34,597 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 1a5268f22ea5dbabf8eac37708fa245e, server=jenkins-hbase4.apache.org,37593,1682362415680 in 1.8330 sec 2023-04-24 18:56:34,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-24 18:56:34,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a5268f22ea5dbabf8eac37708fa245e, UNASSIGN in 1.8370 sec 2023-04-24 18:56:34,598 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682362594598"}]},"ts":"1682362594598"} 2023-04-24 18:56:34,599 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 18:56:34,600 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 18:56:34,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8510 sec 2023-04-24 18:56:34,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 18:56:34,857 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-24 18:56:34,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-24 18:56:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,860 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-24 18:56:34,861 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,864 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:34,866 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C, FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits] 2023-04-24 18:56:34,869 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/6650a887f6b94cbfac6ba3155505440e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/6650a887f6b94cbfac6ba3155505440e 2023-04-24 18:56:34,870 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/77c7a74f02cb44d3bd34fdbb08a4d189 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/77c7a74f02cb44d3bd34fdbb08a4d189 2023-04-24 18:56:34,871 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/db069d0f492946908428a687b11b99ba to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/A/db069d0f492946908428a687b11b99ba 2023-04-24 18:56:34,874 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/88b3abb6c8e34a9185d007cbcafa0a6d to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/88b3abb6c8e34a9185d007cbcafa0a6d 2023-04-24 18:56:34,874 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/bb0ef48c90244e0c8b46a47271f3f0d5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/bb0ef48c90244e0c8b46a47271f3f0d5 2023-04-24 18:56:34,875 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f3bd0ae0ead648b2971599a800bae0a5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/B/f3bd0ae0ead648b2971599a800bae0a5 2023-04-24 18:56:34,877 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5075349587f04b97a4892fbf3f733086 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/5075349587f04b97a4892fbf3f733086 2023-04-24 18:56:34,878 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/6551617b867b4b98a6eb9479be302dd8 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/6551617b867b4b98a6eb9479be302dd8 2023-04-24 18:56:34,879 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/862065d9137d4e37805487356d334c0e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/C/862065d9137d4e37805487356d334c0e 2023-04-24 18:56:34,881 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits/550.seqid to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e/recovered.edits/550.seqid 2023-04-24 18:56:34,881 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/.tmp/data/default/TestAcidGuarantees/1a5268f22ea5dbabf8eac37708fa245e 2023-04-24 18:56:34,881 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 18:56:34,882 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:56:34,882 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 18:56:34,885 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424018d442d4fa846ffa42d6aeb8543a7e5 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424018d442d4fa846ffa42d6aeb8543a7e5 2023-04-24 18:56:34,886 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240b86c9471afb41de953975cb9c90dbcc to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240b86c9471afb41de953975cb9c90dbcc 2023-04-24 18:56:34,886 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424323ed58384234d6abdf736446bc0f145 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424323ed58384234d6abdf736446bc0f145 2023-04-24 18:56:34,887 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444b13d225cb64313a5e3d7a70a1a973f to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444b13d225cb64313a5e3d7a70a1a973f 2023-04-24 18:56:34,888 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424524b5fcc259c40198aabbbf3a11eb6d1 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424524b5fcc259c40198aabbbf3a11eb6d1 2023-04-24 18:56:34,889 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245b147fd0a4c84071b5aae601951c2a55 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245b147fd0a4c84071b5aae601951c2a55 2023-04-24 18:56:34,890 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e35b125c15c4464becdf442ead9e067 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245e35b125c15c4464becdf442ead9e067 2023-04-24 18:56:34,891 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246ab4ba1123364039ac95cfc0c38fb349 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246ab4ba1123364039ac95cfc0c38fb349 2023-04-24 18:56:34,892 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424738ef9a8e66b4b13a862c7b1a9650726 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424738ef9a8e66b4b13a862c7b1a9650726 2023-04-24 18:56:34,893 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424790a83bc1c5340d2a571ce6d6f8b7fcb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424790a83bc1c5340d2a571ce6d6f8b7fcb 2023-04-24 18:56:34,893 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042480fa83a55b4c4dc987ef6a8fbddf0ed3 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042480fa83a55b4c4dc987ef6a8fbddf0ed3 2023-04-24 18:56:34,894 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248e4c49d245094201ad1ef695d20baccb to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304248e4c49d245094201ad1ef695d20baccb 2023-04-24 18:56:34,895 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b6a9f7f46774c458d7f7f27228de1d6 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304249b6a9f7f46774c458d7f7f27228de1d6 2023-04-24 18:56:34,896 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a4a42e794d9b419da392813adde73387 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a4a42e794d9b419da392813adde73387 2023-04-24 18:56:34,897 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bae273b17d4b4b36af790358ef2aea6a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bae273b17d4b4b36af790358ef2aea6a 2023-04-24 18:56:34,898 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd40c1e392d44519ae76ab45a4afc521 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd40c1e392d44519ae76ab45a4afc521 2023-04-24 18:56:34,899 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424beeafdcca6a3483c9432c18916f7d01c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424beeafdcca6a3483c9432c18916f7d01c 2023-04-24 18:56:34,900 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cbadc660943c43acb5ae0bd765a12f17 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424cbadc660943c43acb5ae0bd765a12f17 2023-04-24 18:56:34,901 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d24b9c414a1d4efe9a9c617027c23e4c to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d24b9c414a1d4efe9a9c617027c23e4c 2023-04-24 18:56:34,902 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7de065847684ebe98ca2ebe5c2cc587 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d7de065847684ebe98ca2ebe5c2cc587 2023-04-24 18:56:34,903 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dba28ee64cd841e0a410fddcf0d56b4a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dba28ee64cd841e0a410fddcf0d56b4a 2023-04-24 18:56:34,903 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ddecc5e46f4747d1bc9cdd0fd82e4911 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ddecc5e46f4747d1bc9cdd0fd82e4911 2023-04-24 18:56:34,904 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ede54afae93944ee8178845436d3644e to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ede54afae93944ee8178845436d3644e 2023-04-24 18:56:34,905 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ef1692979e0745c4a5ddeb5633503e6a to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ef1692979e0745c4a5ddeb5633503e6a 2023-04-24 18:56:34,906 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f527a52dff3f4a0491c63c5af1f39763 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f527a52dff3f4a0491c63c5af1f39763 2023-04-24 18:56:34,907 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fe56c2af616a48829b0c0dca6c9a9a43 to hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fe56c2af616a48829b0c0dca6c9a9a43 2023-04-24 18:56:34,907 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 18:56:34,909 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,910 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 18:56:34,912 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 18:56:34,913 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,913 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 18:56:34,913 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682362594913"}]},"ts":"9223372036854775807"} 2023-04-24 18:56:34,914 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 18:56:34,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a5268f22ea5dbabf8eac37708fa245e, NAME => 'TestAcidGuarantees,,1682362569530.1a5268f22ea5dbabf8eac37708fa245e.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 18:56:34,914 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 18:56:34,915 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682362594914"}]},"ts":"9223372036854775807"} 2023-04-24 18:56:34,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 18:56:34,917 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 18:56:34,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-04-24 18:56:34,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45309] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-24 18:56:34,962 INFO [Listener at localhost/43631] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-24 18:56:34,973 INFO [Listener at localhost/43631] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=231 (was 227) - Thread LEAK? -, OpenFileDescriptor=550 (was 563), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=434 (was 376) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4692 (was 4578) - AvailableMemoryMB LEAK? - 2023-04-24 18:56:34,973 INFO [Listener at localhost/43631] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-24 18:56:34,974 INFO [Listener at localhost/43631] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 18:56:34,974 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e324982 to 127.0.0.1:59583 2023-04-24 18:56:34,974 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:34,974 DEBUG [Listener at localhost/43631] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-24 18:56:34,974 DEBUG [Listener at localhost/43631] util.JVMClusterUtil(257): Found active master hash=69527353, stopped=false 2023-04-24 18:56:34,974 INFO [Listener at localhost/43631] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:56:34,979 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-24 18:56:34,979 INFO [Listener at localhost/43631] procedure2.ProcedureExecutor(629): Stopping 2023-04-24 18:56:34,979 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-24 18:56:34,979 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:56:34,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 18:56:34,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 18:56:34,980 DEBUG [Listener at localhost/43631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14082f89 to 127.0.0.1:59583 2023-04-24 18:56:34,980 DEBUG [Listener at localhost/43631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:34,980 INFO [Listener at localhost/43631] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37593,1682362415680' ***** 2023-04-24 18:56:34,980 INFO [Listener at localhost/43631] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-24 18:56:34,981 INFO [RS:0;jenkins-hbase4:37593] regionserver.HeapMemoryManager(220): Stopping 2023-04-24 18:56:34,981 INFO [RS:0;jenkins-hbase4:37593] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-24 18:56:34,981 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-24 18:56:34,981 INFO [RS:0;jenkins-hbase4:37593] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-24 18:56:34,981 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(3304): Received CLOSE for 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:56:34,981 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:34,981 DEBUG [RS:0;jenkins-hbase4:37593] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x612da705 to 127.0.0.1:59583 2023-04-24 18:56:34,982 DEBUG [RS:0;jenkins-hbase4:37593] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:34,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4e0aadbf319324c8ff606d0c7af7f8ed, disabling compactions & flushes 2023-04-24 18:56:34,982 INFO [RS:0;jenkins-hbase4:37593] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-24 18:56:34,982 INFO [RS:0;jenkins-hbase4:37593] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-24 18:56:34,982 INFO [RS:0;jenkins-hbase4:37593] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-24 18:56:34,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:56:34,982 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-24 18:56:34,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:56:34,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. after waiting 0 ms 2023-04-24 18:56:34,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:56:34,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4e0aadbf319324c8ff606d0c7af7f8ed 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-24 18:56:34,983 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-24 18:56:34,983 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 4e0aadbf319324c8ff606d0c7af7f8ed=hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed.} 2023-04-24 18:56:34,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-24 18:56:34,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-24 18:56:34,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-24 18:56:34,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-24 18:56:34,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-24 18:56:34,984 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-24 18:56:34,985 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1505): Waiting on 1588230740, 4e0aadbf319324c8ff606d0c7af7f8ed 2023-04-24 18:56:34,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/.tmp/info/06198c6c99ab4f68ac3c4d554cd2ee81 2023-04-24 18:56:34,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/.tmp/info/06198c6c99ab4f68ac3c4d554cd2ee81 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/info/06198c6c99ab4f68ac3c4d554cd2ee81 2023-04-24 18:56:34,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/info/06198c6c99ab4f68ac3c4d554cd2ee81, entries=2, sequenceid=6, filesize=4.8 K 2023-04-24 18:56:34,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4e0aadbf319324c8ff606d0c7af7f8ed in 15ms, sequenceid=6, compaction requested=false 2023-04-24 18:56:35,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/namespace/4e0aadbf319324c8ff606d0c7af7f8ed/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-24 18:56:35,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:56:35,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4e0aadbf319324c8ff606d0c7af7f8ed: 2023-04-24 18:56:35,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682362417977.4e0aadbf319324c8ff606d0c7af7f8ed. 2023-04-24 18:56:35,034 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-24 18:56:35,034 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-24 18:56:35,046 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-24 18:56:35,185 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-24 18:56:35,386 DEBUG [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-24 18:56:35,391 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/info/c7f79a247eea45af84aa834ae877132d 2023-04-24 18:56:35,394 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c7f79a247eea45af84aa834ae877132d 2023-04-24 18:56:35,399 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/rep_barrier/f2b3d72e028e47d9b156e634382cec79 2023-04-24 18:56:35,402 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f2b3d72e028e47d9b156e634382cec79 2023-04-24 18:56:35,407 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/table/f701675d4fe8408b91b89aeedb81275c 2023-04-24 18:56:35,410 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f701675d4fe8408b91b89aeedb81275c 2023-04-24 18:56:35,410 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/info/c7f79a247eea45af84aa834ae877132d as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/info/c7f79a247eea45af84aa834ae877132d 2023-04-24 18:56:35,413 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c7f79a247eea45af84aa834ae877132d 2023-04-24 18:56:35,413 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/info/c7f79a247eea45af84aa834ae877132d, entries=22, sequenceid=93, filesize=7.1 K 2023-04-24 18:56:35,414 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/rep_barrier/f2b3d72e028e47d9b156e634382cec79 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/rep_barrier/f2b3d72e028e47d9b156e634382cec79 2023-04-24 18:56:35,417 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f2b3d72e028e47d9b156e634382cec79 2023-04-24 18:56:35,417 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/rep_barrier/f2b3d72e028e47d9b156e634382cec79, entries=6, sequenceid=93, filesize=5.5 K 2023-04-24 18:56:35,417 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/.tmp/table/f701675d4fe8408b91b89aeedb81275c as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/table/f701675d4fe8408b91b89aeedb81275c 2023-04-24 18:56:35,419 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f701675d4fe8408b91b89aeedb81275c 2023-04-24 18:56:35,420 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/table/f701675d4fe8408b91b89aeedb81275c, entries=9, sequenceid=93, filesize=5.5 K 2023-04-24 18:56:35,420 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 436ms, sequenceid=93, compaction requested=false 2023-04-24 18:56:35,427 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-24 18:56:35,427 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-24 18:56:35,428 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-24 18:56:35,428 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-24 18:56:35,428 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-24 18:56:35,586 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37593,1682362415680; all regions closed. 2023-04-24 18:56:35,594 DEBUG [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/oldWALs 2023-04-24 18:56:35,595 INFO [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37593%2C1682362415680.meta:.meta(num 1682362417701) 2023-04-24 18:56:35,598 DEBUG [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/oldWALs 2023-04-24 18:56:35,599 INFO [RS:0;jenkins-hbase4:37593] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37593%2C1682362415680:(num 1682362417176) 2023-04-24 18:56:35,599 DEBUG [RS:0;jenkins-hbase4:37593] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:35,599 INFO [RS:0;jenkins-hbase4:37593] regionserver.LeaseManager(133): Closed leases 2023-04-24 18:56:35,599 INFO [RS:0;jenkins-hbase4:37593] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-24 18:56:35,599 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-24 18:56:35,601 INFO [RS:0;jenkins-hbase4:37593] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37593 2023-04-24 18:56:35,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 18:56:35,606 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37593,1682362415680 2023-04-24 18:56:35,606 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6b60e1ef rejected from java.util.concurrent.ThreadPoolExecutor@6f5f685b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2477] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,607 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 18:56:35,607 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@773e0bef rejected from java.util.concurrent.ThreadPoolExecutor@6f5f685b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2477] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,607 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37593,1682362415680] 2023-04-24 18:56:35,608 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37593,1682362415680; numProcessing=1 2023-04-24 18:56:35,609 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37593,1682362415680 already deleted, retry=false 2023-04-24 18:56:35,609 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37593,1682362415680 expired; onlineServers=0 2023-04-24 18:56:35,609 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45309,1682362414518' ***** 2023-04-24 18:56:35,609 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-24 18:56:35,609 DEBUG [M:0;jenkins-hbase4:45309] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4519ed02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-24 18:56:35,609 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45309,1682362414518 2023-04-24 18:56:35,609 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45309,1682362414518; all regions closed. 2023-04-24 18:56:35,609 DEBUG [M:0;jenkins-hbase4:45309] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 18:56:35,610 DEBUG [M:0;jenkins-hbase4:45309] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-24 18:56:35,610 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-24 18:56:35,610 DEBUG [M:0;jenkins-hbase4:45309] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-24 18:56:35,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682362416811] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682362416811,5,FailOnTimeoutGroup] 2023-04-24 18:56:35,610 INFO [M:0;jenkins-hbase4:45309] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-24 18:56:35,610 INFO [M:0;jenkins-hbase4:45309] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-24 18:56:35,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682362416812] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682362416812,5,FailOnTimeoutGroup] 2023-04-24 18:56:35,610 INFO [M:0;jenkins-hbase4:45309] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-24 18:56:35,610 DEBUG [M:0;jenkins-hbase4:45309] master.HMaster(1512): Stopping service threads 2023-04-24 18:56:35,610 INFO [M:0;jenkins-hbase4:45309] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-24 18:56:35,611 ERROR [M:0;jenkins-hbase4:45309] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-24 18:56:35,611 INFO [M:0;jenkins-hbase4:45309] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-24 18:56:35,611 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-24 18:56:35,612 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-24 18:56:35,612 DEBUG [M:0;jenkins-hbase4:45309] zookeeper.ZKUtil(398): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-24 18:56:35,612 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 18:56:35,612 WARN [M:0;jenkins-hbase4:45309] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-24 18:56:35,612 INFO [M:0;jenkins-hbase4:45309] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-24 18:56:35,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 18:56:35,612 INFO [M:0;jenkins-hbase4:45309] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-24 18:56:35,613 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-24 18:56:35,613 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:56:35,613 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:56:35,613 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-24 18:56:35,613 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:56:35,613 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.77 KB heapSize=392.89 KB 2023-04-24 18:56:35,629 INFO [M:0;jenkins-hbase4:45309] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.77 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ba2f46ccd5c042358d5aa3a7d83cb891 2023-04-24 18:56:35,633 INFO [M:0;jenkins-hbase4:45309] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ba2f46ccd5c042358d5aa3a7d83cb891 2023-04-24 18:56:35,634 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ba2f46ccd5c042358d5aa3a7d83cb891 as hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ba2f46ccd5c042358d5aa3a7d83cb891 2023-04-24 18:56:35,638 INFO [M:0;jenkins-hbase4:45309] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ba2f46ccd5c042358d5aa3a7d83cb891 2023-04-24 18:56:35,638 INFO [M:0;jenkins-hbase4:45309] regionserver.HStore(1080): Added hdfs://localhost:36683/user/jenkins/test-data/377de52e-3045-285f-8ffc-0d385fadcbed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ba2f46ccd5c042358d5aa3a7d83cb891, entries=65, sequenceid=570, filesize=24.7 K 2023-04-24 18:56:35,638 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegion(2948): Finished flush of dataSize ~344.77 KB/353044, heapSize ~392.88 KB/402304, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 25ms, sequenceid=570, compaction requested=false 2023-04-24 18:56:35,639 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 18:56:35,639 DEBUG [M:0;jenkins-hbase4:45309] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 18:56:35,642 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-24 18:56:35,642 INFO [M:0;jenkins-hbase4:45309] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-24 18:56:35,643 INFO [M:0;jenkins-hbase4:45309] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45309 2023-04-24 18:56:35,644 DEBUG [M:0;jenkins-hbase4:45309] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45309,1682362414518 already deleted, retry=false 2023-04-24 18:56:35,708 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 18:56:35,708 INFO [RS:0;jenkins-hbase4:37593] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37593,1682362415680; zookeeper connection closed. 2023-04-24 18:56:35,709 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@fb45df2 rejected from java.util.concurrent.ThreadPoolExecutor@6f5f685b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2477] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,709 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): regionserver:37593-0x101b7851c360001, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 18:56:35,709 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1b4e0f4f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1b4e0f4f 2023-04-24 18:56:35,709 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4d141f49 rejected from java.util.concurrent.ThreadPoolExecutor@6f5f685b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2477] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,710 INFO [Listener at localhost/43631] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-24 18:56:35,808 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 18:56:35,808 INFO [M:0;jenkins-hbase4:45309] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45309,1682362414518; zookeeper connection closed. 2023-04-24 18:56:35,809 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@44d39f93 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4145] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,809 DEBUG [Listener at localhost/43631-EventThread] zookeeper.ZKWatcher(600): master:45309-0x101b7851c360000, quorum=127.0.0.1:59583, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 18:56:35,809 ERROR [Listener at localhost/43631-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6de45497 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4145] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 18:56:35,811 WARN [Listener at localhost/43631] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-24 18:56:35,814 INFO [Listener at localhost/43631] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-24 18:56:35,922 WARN [BP-1841710233-172.31.14.131-1682362411180 heartbeating to localhost/127.0.0.1:36683] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-24 18:56:35,923 WARN [BP-1841710233-172.31.14.131-1682362411180 heartbeating to localhost/127.0.0.1:36683] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1841710233-172.31.14.131-1682362411180 (Datanode Uuid 45dafbf2-b08d-4ede-9eaf-5017a3e0a134) service to localhost/127.0.0.1:36683 2023-04-24 18:56:35,925 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7/dfs/data/data1/current/BP-1841710233-172.31.14.131-1682362411180] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-24 18:56:35,926 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c93c1a16-04d2-241c-5c8f-2f8469b41640/cluster_01634698-f22e-0f54-453d-bae6c0268df7/dfs/data/data2/current/BP-1841710233-172.31.14.131-1682362411180] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-24 18:56:35,968 INFO [Listener at localhost/43631] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-24 18:56:36,131 INFO [Listener at localhost/43631] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-24 18:56:36,317 INFO [Listener at localhost/43631] hbase.HBaseTestingUtility(1293): Minicluster is down